diff --git a/docs/next/public/sitemap.xml b/docs/next/public/sitemap.xml index ba50d81abc4af..fe8b32fec8cef 100644 --- a/docs/next/public/sitemap.xml +++ b/docs/next/public/sitemap.xml @@ -49,6 +49,10 @@ https://docs.dagster.io/docs/apidocs/libraries/dagster_dask + + https://docs.dagster.io/docs/apidocs/libraries/dagster_databricks + + https://docs.dagster.io/docs/apidocs/libraries/dagster_datadog diff --git a/docs/next/src/treeOfContents.json b/docs/next/src/treeOfContents.json index d59e4b341b114..dde8fad6dc628 100644 --- a/docs/next/src/treeOfContents.json +++ b/docs/next/src/treeOfContents.json @@ -164,6 +164,10 @@ "name": "Datadog (dagster_datadog)", "path": "/docs/apidocs/libraries/dagster_datadog" }, + { + "name": "Databricks (dagster_databricks)", + "path": "/docs/apidocs/libraries/dagster_databricks" + }, { "name": "DBT (dagster_dbt)", "path": "/docs/apidocs/libraries/dagster_dbt" diff --git a/docs/sections/api/apidocs/libraries/dagster_databricks.rst b/docs/sections/api/apidocs/libraries/dagster_databricks.rst new file mode 100644 index 0000000000000..d73040cbbc929 --- /dev/null +++ b/docs/sections/api/apidocs/libraries/dagster_databricks.rst @@ -0,0 +1,26 @@ +Databricks (dagster_databricks) +------------------------- + +The ``dagster_databricks`` package provides two main pieces of functionality: + +- a resource, ``databricks_pyspark_step_launcher``, which will execute a solid within a Databricks + context on a cluster, such that the ``pyspark`` resource uses the cluster's Spark instance; and +- a solid, ``DatabricksRunJobSolidDefinition``, which submits an external configurable job to + Databricks using the 'Run Now' API. + +See the 'simple_pyspark' Dagster example for an example of how to use the resource. + +Note that either S3 or Azure Data Lake Storage config **must** be specified for solids to succeed, +and the credentials for this storage must also be stored as a Databricks Secret and stored in the +resource config so that the Databricks cluster can access storage. + +.. currentmodule:: dagster_databricks + +.. autodata:: dagster_databricks.databricks_pyspark_step_launcher + :annotation: ResourceDefinition + +.. autoclass:: dagster_databricks.DatabricksRunJobSolidDefinition + +.. autoclass:: dagster_databricks.DatabricksJobRunner + +.. autoclass:: dagster_databricks.DatabricksError