Add API docs for dagster_databricks

docs/next/public/sitemap.xml

@@ -49,6 +49,10 @@
         <loc>https://docs.dagster.io/docs/apidocs/libraries/dagster_dask</loc>
   </url>
 
+  <url>
+        <loc>https://docs.dagster.io/docs/apidocs/libraries/dagster_databricks</loc>
+  </url>
+
   <url>
         <loc>https://docs.dagster.io/docs/apidocs/libraries/dagster_datadog</loc>
   </url>

docs/next/src/treeOfContents.json

@@ -164,6 +164,10 @@
         "name": "Datadog (dagster_datadog)",
         "path": "/docs/apidocs/libraries/dagster_datadog"
       },
+      {
+        "name": "Databricks (dagster_databricks)",
+        "path": "/docs/apidocs/libraries/dagster_databricks"
+      },
       {
         "name": "DBT (dagster_dbt)",
         "path": "/docs/apidocs/libraries/dagster_dbt"

docs/sections/api/apidocs/libraries/dagster_databricks.rst

@@ -0,0 +1,26 @@
+Databricks (dagster_databricks)
+-------------------------
+
+The ``dagster_databricks`` package provides two main pieces of functionality:
+
+- a resource, ``databricks_pyspark_step_launcher``, which will execute a solid within a Databricks
+  context on a cluster, such that the ``pyspark`` resource uses the cluster's Spark instance; and
+- a solid, ``DatabricksRunJobSolidDefinition``, which submits an external configurable job to
+  Databricks using the 'Run Now' API.
+
+See the 'simple_pyspark' Dagster example for an example of how to use the resource.
+
+Note that either S3 or Azure Data Lake Storage config **must** be specified for solids to succeed,
+and the credentials for this storage must also be stored as a Databricks Secret and stored in the
+resource config so that the Databricks cluster can access storage.
+
+.. currentmodule:: dagster_databricks
+
+.. autodata:: dagster_databricks.databricks_pyspark_step_launcher
+  :annotation: ResourceDefinition
+
+.. autoclass:: dagster_databricks.DatabricksRunJobSolidDefinition
+
+.. autoclass:: dagster_databricks.DatabricksJobRunner
+
+.. autoclass:: dagster_databricks.DatabricksError