Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Reapply Databricks agent docs changes from #4008 #4751

Merged
merged 1 commit into from
Jan 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/deployment/agents/bigquery.rst
Original file line number Diff line number Diff line change
Expand Up @@ -102,3 +102,5 @@ Upgrade the Flyte Helm release
Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte``)

and ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``).

For BigQuery plugin on the Flyte cluster, please refer to `BigQuery Plugin Example <https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_plugin/bigquery.html>`_
294 changes: 294 additions & 0 deletions docs/deployment/agents/databricks.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,294 @@
.. _deployment-agent-setup-databricks:

Databricks Agent
=================

This guide provides an overview of how to set up Databricks agent in your Flyte deployment.

Spin up a cluster
-----------------

.. tabs::

.. group-tab:: Flyte binary

You can spin up a demo cluster using the following command:

.. code-block:: bash

flytectl demo start

Or install Flyte using the :ref:`flyte-binary helm chart <deployment-deployment-cloud-simple>`.

.. group-tab:: Flyte core

If you've installed Flyte using the
`flyte-core helm chart <https://github.com/flyteorg/flyte/tree/master/charts/flyte-core>`__, please ensure:

* You have the correct kubeconfig and have selected the correct Kubernetes context.
* You have configured the correct flytectl settings in ``~/.flyte/config.yaml``.

.. note::

Add the Flyte chart repo to Helm if you're installing via the Helm charts.

.. code-block:: bash

helm repo add flyteorg https://flyteorg.github.io/flyte

Databricks workspace
--------------------

To set up your Databricks account, follow these steps:

1. Create a `Databricks account <https://www.databricks.com/>`__.

.. image:: https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/deployment/plugins/databricks/databricks_workspace.png
:alt: A screenshot of Databricks workspace creation.

2. Ensure that you have a Databricks workspace up and running.

.. image:: https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/deployment/plugins/databricks/open_workspace.png
:alt: A screenshot of Databricks workspace.

3. Generate a `personal access token
<https://docs.databricks.com/dev-tools/auth.html#databricks-personal-ACCESS_TOKEN-authentication>`__ to be used in the Flyte configuration.
You can find the personal access token in the user settings within the workspace. ``User settings`` -> ``Developer`` -> ``Access tokens``

.. image:: https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/deployment/plugins/databricks/databricks_access_token.png
:alt: A screenshot of access token.

4. Enable custom containers on your Databricks cluster before you trigger the workflow.

.. code-block:: bash

curl -X PATCH -n -H "Authorization: Bearer <your-personal-access-token>" \
https://<databricks-instance>/api/2.0/workspace-conf \
-d '{"enableDcs": "true"}'

For more detail, check `custom containers <https://docs.databricks.com/administration-guide/clusters/container-services.html>`__.

5. Create an `instance profile
<https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html>`__
for the Spark cluster. This profile enables the Spark job to access your data in the S3 bucket.

Create an instance profile using the AWS console (For AWS Users)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

1. In the AWS console, go to the IAM service.
2. Click the Roles tab in the sidebar.
3. Click Create role.

a. Under Trusted entity type, select AWS service.
b. Under Use case, select **EC2**.
c. Click Next.
d. At the bottom of the page, click Next.
e. In the Role name field, type a role name.
f. Click Create role.

4. In the role list, click the **AmazonS3FullAccess** role.
5. Click Create role button.

In the role summary, copy the Role ARN.

.. image:: https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/deployment/plugins/databricks/s3_arn.png
:alt: A screenshot of s3 arn.

Locate the IAM role that created the Databricks deployment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you don’t know which IAM role created the Databricks deployment, do the following:

1. As an account admin, log in to the account console.
2. Go to ``Workspaces`` and click your workspace name.
3. In the Credentials box, note the role name at the end of the Role ARN

For example, in the Role ARN ``arn:aws:iam::123456789123:role/finance-prod``, the role name is finance-prod

Edit the IAM role that created the Databricks deployment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. In the AWS console, go to the IAM service.
2. Click the Roles tab in the sidebar.
3. Click the role that created the Databricks deployment.
4. On the Permissions tab, click the policy.
5. Click Edit Policy.
6. Append the following block to the end of the Statement array. Ensure that you don’t overwrite any of the existing policy. Replace <iam-role-for-s3-access> with the role you created in Configure S3 access with instance profiles.

.. code-block:: bash

{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::<aws-account-id-databricks>:role/<iam-role-for-s3-access>"
}

Specify agent configuration
----------------------------

.. tabs::

.. group-tab:: Flyte binary

.. tabs::

.. group-tab:: Demo cluster

Enable the Databricks agent on the demo cluster by updating the ConfigMap:

.. code-block:: bash

kubectl edit configmap flyte-sandbox-config -n flyte

.. code-block:: yaml
:emphasize-lines: 7,12,16

tasks:
task-plugins:
default-for-task-types:
container: container
container_array: k8s-array
sidecar: sidecar
spark: agent-service
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
plugins:
agent-service:
supportedTaskTypes:
- spark

.. group-tab:: Helm chart

Edit the relevant YAML file to specify the plugin.

.. code-block:: yaml
:emphasize-lines: 7,11,15

tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
- container: container
- container_array: k8s-array
- spark: agent-service
plugins:
agent-service:
supportedTaskTypes:
- spark

.. group-tab:: Flyte core

Create a file named ``values-override.yaml`` and add the following config to it:

.. code-block:: yaml
:emphasize-lines: 9,14-17

enabled_plugins:
tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
container: container
sidecar: sidecar
container_array: k8s-array
spark: agent-service
plugins:
agent-service:
supportedTaskTypes:
- spark

Add the Databricks access token
-------------------------------

You have to set the Databricks token to the Flyte configuration.

1. Install flyteagent pod using helm

.. code-block::

helm repo add flyteorg https://flyteorg.github.io/flyte
helm install flyteagent flyteorg/flyteagent --namespace flyte

2. Get the base64 value of your Databricks token.

.. code-block::

echo -n "<DATABRICKS_TOKEN>" | base64

3. Edit the flyteagent secret

.. code-block:: bash

kubectl edit secret flyteagent -n flyte

.. code-block:: yaml
:emphasize-lines: 3

apiVersion: v1
data:
flyte_databricks_access_token: <BASE64_ENCODED_DATABRICKS_TOKEN>
kind: Secret
metadata:
annotations:
meta.helm.sh/release-name: flyteagent
meta.helm.sh/release-namespace: flyte
creationTimestamp: "2023-10-04T04:09:03Z"
labels:
app.kubernetes.io/managed-by: Helm
name: flyteagent
namespace: flyte
resourceVersion: "753"
uid: 5ac1e1b6-2a4c-4e26-9001-d4ba72c39e54
type: Opaque


Upgrade the deployment
----------------------

.. tabs::

.. group-tab:: Flyte binary

.. tabs::

.. group-tab:: Demo cluster

.. code-block::

kubectl rollout restart deployment flyte-sandbox -n flyte

.. group-tab:: Helm chart

.. code-block::

helm upgrade <RELEASE_NAME> flyteorg/flyte-binary -n <YOUR_NAMESPACE> --values <YOUR_YAML_FILE>

Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte-backend``),
``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``),
and ``<YOUR_YAML_FILE>`` with the name of your YAML file.

.. group-tab:: Flyte core

.. code-block::

helm upgrade <RELEASE_NAME> flyte/flyte-core -n <YOUR_NAMESPACE> --values values-override.yaml

Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte``)
and ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``).

Wait for the upgrade to complete. You can check the status of the deployment pods by running the following command:

.. code-block::

kubectl get pods -n flyte

For databricks plugin on the Flyte cluster, please refer to `Databricks Plugin Example <https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_plugin/index.html>`_
3 changes: 3 additions & 0 deletions docs/deployment/agents/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ Discover the process of setting up Agents for Flyte.
- Guide to setting up the MMCloud agent.
* - {ref}`Sensor Agent <deployment-agent-setup-sensor>`
- Guide to setting up the Sensor agent.
* - {ref} `Databricks Agent <deployment-agent-setup-databricks>`
- Guide to setting up the Databricks agent.
```

```{toctree}
Expand All @@ -25,5 +27,6 @@ Discover the process of setting up Agents for Flyte.

bigquery
mmcloud
databricks
sensor
```
8 changes: 5 additions & 3 deletions docs/deployment/agents/mmcloud.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,15 @@ Enable the MMCloud agent by adding the following config to the relevant YAML fil
.. code-block:: yaml

tasks:
task-agents:
enabled-agents:
task-plugins:
enabled-plugins:
- agent-service
default-for-task-types:
- mmcloud_task: agent-service

.. code-block:: yaml

agents:
plugins:
agent-service:
agents:
mmcloud-agent:
Expand Down Expand Up @@ -117,3 +117,5 @@ Wait for the upgrade to complete. You can check the status of the deployment pod
.. code-block::

kubectl get pods -n flyte

For MMCloud plugin on the Flyte cluster, please refer to `Memory Machine Cloud Plugin Example <https://docs.flyte.org/en/latest/flytesnacks/examples/mmcloud_plugin/index.html>`_
4 changes: 3 additions & 1 deletion docs/deployment/plugins/webapi/snowflake.rst
Original file line number Diff line number Diff line change
Expand Up @@ -239,4 +239,6 @@ Wait for the upgrade to complete. You can check the status of the deployment pod

.. code-block::

kubectl get pods -n flyte
kubectl get pods -n flyte

For Snowflake plugin on the Flyte cluster, please refer to `Snowflake Plugin Example <https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_plugin/snowflake.html>`_
Loading