Integration with Pyspark

[jacksongoode]

Just wondering how one would go about integrating this project within a Pyspark environment? We are also using Databricks. What configuration would be needed? Just the jar file? Thanks!

[nevi-me]

Hey @jacksongoode. I've managed to get blaze running locally with pyspark, as well as on a Databricks cluster.

For Databricks, you'd need to do the following:

• Set up a cluster with Spark 3.0 as Blaze isn't yet built against 3.2 (Spark 3.2 Support #168)
• Check what Ubuntu version the cluster above runs on, it should be Ubuntu 18.04 LTS. You'll need the same glibc version, so this is important.
• Build the Blaze jar on the same Ubuntu version as above. Docker's fine, I used WSL at the time.
• Upload the jar somewhere in the FileStore, or install it as a library just so it can be copied to dbfs, note the location where it's stored.
• Create a Databricks init script, and copy the jar to the Databricks jars as part of the script. You need to do this because the jar must exist when the cluster starts. The libraries only get installed after startup, which is too late. Some googling will help, else I can provide screenshots when I boot up a cluster again, was playing around in my spare time.
• Set the spark.org.extensions and shuffle manager configs as per the README.
• Restart cluster

Once you do this the first time, it becomes quicker to update the jar for new changes.

You probably won't be able to run (m)any queries though, as you'll encounter #174 when trying to read even plain parquet files from the cluster.

For a local setup, you can copy the jar into the spark jars folder, then set the config as part of spark.getOrCreate().

[bitscum]

Gonna be keeping an eye on this! I'm using Databricks too. Using Blaze could be very interesting in the future.

[jacksongoode]

@nevi-me Hi Nevi, it's been a minute! Love to know if these instructions still hold or if it's gotten easier?