Databricks-Connect Container

This container is designed for developing PySpark application in VS Code using Databricks-Connect. Technically it should work for Scala, Java & R - though I haven't tried.

It can be used as a Docker Container or as a cloud hosted container using CodeSpaces.

Requirements

Mac OS

• VS Code with the Remote Containers Extension
• Docker or a CodeSpace Plan

Windows 10

• VS Code with the Remote Containers Extension
• Docker or a CodeSpace Plan
• If using Docker:
  • WSL or WSL2 (WSL2 Preferred)

Alternative

For a more advanced option you can use any OS and a Remote Docker Host.

This maybe useful if you cannot run local containers and do not want to use a cloud container.

Getting Started

• Open an empty folder in VS Code
• Create a directory called .devcontainer
• Create an empty file in the directory called devcontainer.json
• Paste this code into the file:

{
	"context": "..",
	"image": "datathirstltd/dbconnect:7.1.0",

	"settings": {
		"python.pythonPath": "/opt/conda/envs/dbconnect/bin/python",
		"python.venvPath": "/opt/conda/envs/dbconnect/lib/python3.7/site-packages/pyspark/jars"
	},

	//  Optional command - could add your own environment.yml file here (you must keep --name the same)
	// "postCreateCommand": "conda env update --file environment.yml --name dbconnect",
	
	// Rather than storing/committing your bearer token here we recommend using a local variable and passing thru "DATABRICKS_API_TOKEN": "${localEnv:DatabricksToken}",
	// You can manually set these as environment variables if you prefer
	"containerEnv": {
		"DATABRICKS_ADDRESS": "https://westeurope.azuredatabricks.net/",
		"DATABRICKS_API_TOKEN": "dapia12345678901234567890",
		"DATABRICKS_CLUSTER_ID": "0000-11111-hello123",
		"DATABRICKS_ORG_ID": "1234567890",
		"DATABRICKS_PORT": "8787"
	},
	"extensions": [
		"ms-python.python"
	]
}  

IMPORTANT: Correct the image tag to the version of Databricks Runtime your cluster is running. Currently we only support Databricks 6+

• Update the Databricks Variables for your environment
• Optionally add any additional extensions you want to the extensions block.

IMPORTANT: Changing any setting in the devcontain.json after the container has been build requires you to rebuild the container for it take effect

To open using Docker locally:

• Click on the Green icon in the bottom left of VSCode and select "Reopen in Container"

To open in a CodeSpace:

• Commit your folder to a repo first
• Open the Remote Explorer (left hand toolbar)
• Ensure CodeSpaces is selected in the top drop down
• Click + (Create new CodeSpace)
• Follow the prompts

The first pull can be a little slow as the image is quite big. But once it is cached rebuilding the container should take just a few seconds.

Test it out

First from command prompt check that you databricks connect install can connect:

databricks-connect test

If create a test.py file and paste this code:

from pyspark.sql import SparkSession
spark = SparkSession\
.builder\
.getOrCreate()

print("Testing simple count")

# The Spark code will execute on the Azure Databricks cluster.
print(spark.range(100).count())

Press F5 and select the Python debugger.

Why?

Because setting up Databricks-Connect (particulary on Windows is a PIA). This allow:

• A common setup between team members
• Multiple side by side versions
• Ability to reset your environment
• Even run the whole thing from a browser!

GitHub Repository

Issues & Contributions

https://github.com/DataThirstLtd/databricksConnectDocker

Docker Hub

https://hub.docker.com/r/datathirstltd/dbconnect

More Information

About Data Thirst - https://datathirst.net About VSCode and Containers: https://code.visualstudio.com/docs/remote/containers