deploy microsoft.spark.worker in spark standalone cluste mode on windows.

[msgurikar]

Hi,
Can you please guide me how to run spark/deployment/install-worker.sh to setup microsoft.spark.worker locally on windows 10 machine.

[dbeavon]

Did you go thru a full tutorial? There are a number of steps:

https://docs.microsoft.com/en-us/dotnet/spark/tutorials/get-started?tabs=windows

One thing that was surprising to me is the extent to which environment variables are used. It doesn't feel natural (on Windows) to have such high-stakes variables, but eventually I got used to it. Make sure to pay close attention to the values of DOTNET_WORKER_DIR and DOTNET_ASSEMBLY_SEARCH_PATHS.

... see https://docs.microsoft.com/en-us/dotnet/spark/tutorials/get-started?tabs=windows#7-set-dotnet_worker_dir-and-check-dependencies

[msgurikar]

Thank you for your time.
Yes. I did go through this.
https://docs.microsoft.com/en-us/dotnet/spark/tutorials/get-started?tabs=windows#7-set-dotnet_worker_dir-and-check-dependencies

It mentions setting up master in local mode, i am able to successfully run in local mode with these steps.

We want to run .net spark in cluster mode by setting up master and workers in different windows machines.
http://spark.apache.org/docs/latest/spark-standalone.html#installing-spark-standalone-to-a-cluster

[dbeavon]

@msgurikar Sorry for the delay. I'm fairly new to spark as well. There is often a bit of confusion in the way people use the word "local" in the context of Spark.

The way I use it is only to describe the special configuration of the --master parameter (eg. --master local). This results in an entire spark environment that runs within the context of a single process (ie. the driver and workers will all be hosted together in one place, and no other processes need to be running ahead of time, before issuing the spark-submit).

For example the following command:

spark-submit ^
--class org.apache.spark.deploy.dotnet.DotnetRunner ^
--master local ^
microsoft-spark-3-0_2.12-<version>.jar ^
dotnet MySparkApp.dll <path-of-input.txt>

... will result in some messages that describe the fact that a new cluster is being started in a process, along with its own U/I. You will see messages like so:

21/04/02 14:21:55 INFO Utils: Successfully started service 'SparkUI' on port 4041.

... and you will have a management console available to you at an address like so: http://localhost:4041

That local mode of Spark is running locally (of course) but more than that, it is transient and its entire purpose is scoped to a single spark-submit.

However you can also run a standalone spark cluster locally on a single machine as well. This is different than "--master local". I would recommend trying to get that working. There are a few subtle differences. The biggest is that you need to start the master and worker nodes ahead of time, and they live in their own processes. I'd highly recommend using Process Explorer to review the various java processes that are started when using this configuration.

Standalone spark cluster on Local Machine

First go to your installation directory and start the master.

cd C:\mssql\olap\data\spark-3.0.0-bin-hadoop3.2\bin 
spark-class org.apache.spark.deploy.master.Master

Note that it should tell you what URL to use for accessing the spark U/I, and how to register workers (spark://172.33.11.222:7077) Then after that process is running you can start as many worker processes as needed:

cd C:\mssql\olap\data\spark-3.0.0-bin-hadoop3.2\bin
spark-class org.apache.spark.deploy.worker.Worker  spark://172.33.11.222:7077

Then you are ready to use the standalone spark cluster. You just reference the master in the argument to spark-submit:

spark-submit --master spark://172.33.11.222:7077 --class org.apache.spark.deploy.dotnet.DotnetRunner ...

For all intents and purposes this is a full/normal spark cluster. But all of the related processes will be running on a single machine. Between these two options (standalone cluster configuration, and the "local" mode), I've been able to cover all my bases and develop code that is compatible with Databricks. We don't currently maintain our own production spark clusters on-premises ... aside from what is necessary for development purposes.

I think it would be counter-productive to get a development environment running on multiple machines. It will be harder to attach the debugger, and harder to monitor resource usage (disk, network, cpu, RAM, etc). There really isn't a reason to run the processes remotely during development unless you really think it would fundamentally change the way your logic is behaving. (or you are working with extremely unwieldy datasets in development).

Do you intend on hosting your own spark cluster in production?

[msgurikar]

Hi David,
Thank you so much for the detailed reply. Its very clear. Yes, i am able to run .NEt Spark in Standalone cluster mode by launching spark master and workers as you mentioned above. And i could .NET code using the above submit command, it works well.

Yes, as you mentioned here, we are having development physical cluster setup in which we are running master in one machine and workers in other machine. We are able to run our pyspark job(Python files) successfully, however we are having difficulty in setting up and sending Microsoft.Spark.Worker binaries and our code binaries.
I tried to follow Yarn steps here
https://docs.microsoft.com/en-us/dotnet/spark/how-to-guides/deploy-worker-udf-binaries

--conf spark.yarn.appMasterEnv.DOTNET_WORKER_DIR=./worker/Microsoft.Spark.Worker-
--conf spark.yarn.appMasterEnv.DOTNET_ASSEMBLY_SEARCH_PATHS=./udfs
--archives hdfs:///Microsoft.Spark.Worker.net461.win-x64-.zip#worker,hdfs:///mySparkApp.zip#udfs
hdfs:///microsoft-spark-<spark_majorversion-spark_minorversion>_<scala_majorversion.scala_minorversion>-<spark_dotnet_version>.jar
hdfs:///mySparkApp.zip mySparkApp

However we dont have hdfs setup, and --archives is not working in file: windows shared driver.

Can you please suggest us how to pass/send Microsoft.Spark.Worker and code binaries to workers/executors machine running in windows cluster on different machines.
Thank you.
Sorry for late reply.

[dbeavon]

@msgurikar
I've never used the yarn cluster manager yet, so I can't help much with that. You may want to avoid that for the initial testing. Just do the same thing you did to get a standalone cluster working on a single machine.

FYI, here is an explanation of the difference between yarn and default cluster mode:
https://stackoverflow.com/questions/28664834/which-cluster-type-should-i-choose-for-spark#:~:text=YARN%20allows%20you%20to%20dynamically,frameworks%20that%20run%20on%20YARN.&text=If%20you%20have%20a%20big,nodes%20with%20the%20shared%20secret.

As far as your files and custom assemblies are concerned, I would suggest you host the files on a network drive (NAS). This is probably analogous to "dbfs" in databricks. It will allow the same files to be available to everyone.

You might want to see how others set this up before doing it yourself. You can follow the instructions on how to get code deployed to azure-databricks. By working thru that process, you may be able to learn what you will need to do this in your on-prem environment. And the upshot is that you can open support tickets if you run into any trouble with their instructions. It shouldn't cost much to get started with azure-databricks, probably just a one month subscription, with less than $100 worth of cost.

I think you are on the right track. The two main issues are making sure that the workers are able to find Microsoft.Spark.Worker (in DOTNET_WORKER_DIR) and making sure that they can subsequently load your custom assemblies (in DOTNET_ASSEMBLY_SEARCH_PATHS).

[msgurikar]

Hi David,
I have tried run .NET spark code on dev physical cluster, here are my steps.

created shared network drive and mappe to K:
Added Worker and our debug binaries here

And on 1 of cluster machine i.e XX.XXXX.10.2 , i have launched spark master with these settings
set DOTNET_WORKER_DIR=K:/Microsoft.Spark.Worker-1.0.0
set DOTNET_WORKER_DEBUG=1
set DOTNET_ASSEMBLY_SEARCH_PATHS=K:/dotnetapp_binaries/Debug
set PATH=%SPARK_HOME%\bin;%DOTNET_WORKER_DIR%;%DOTNET_ASSEMBLY_SEARCH_PATHS%;%PATH%

spark-class2.cmd org.apache.spark.deploy.master.Master

And on the same machine, launching spark worker with the same setting using below command
spark-class2.cmd org.apache.spark.deploy.worker.Worker -c 24 -m 72G spark://XX.XXX.10.2:7077

Now if run my code as driver on the same machine with the below command having these settings
set DOTNET_WORKER_DIR=K:/Microsoft.Spark.Worker-1.0.0
set DOTNET_WORKER_DEBUG=1
set DOTNET_ASSEMBLY_SEARCH_PATHS=K:/dotnetapp_binaries/Debug/
set PATH=%SPARK_HOME%\bin;%DOTNET_WORKER_DIR%;%DOTNET_ASSEMBLY_SEARCH_PATHS%;%PATH%

\DotnetSpark.Package\Submit-Job.cmd --class org.apache.spark.deploy.dotnet.DotnetRunner --master spark://10.151.100.2:7077 microsoft-spark-3-0_2.12-1.0.0.jar .\dotnetapp1.exe

It works.

Now, if start this code on another machine(driver) having ip XX.XXXX.10.4 in the same network using this command
C:\ARCH\ARCHimedes\DotnetSpark.Package\Submit-Job.cmd --class org.apache.spark.deploy.dotnet.DotnetRunner --master spark://10.151.100.2:7077 microsoft-spark-3-0_2.12-1.0.0.jar .\dotnetapp1.exe

It doesn't work and when i see the worker log, i get unable to connect error
Caused by: java.io.IOException: Failed to connect to host.docker.internal/xx.xxx.10.2:62270
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:253)

Even i tried with this command,
C:\ARCH\ARCHimedes\DotnetSpark.Package\Submit-Job.cmd --class org.apache.spark.deploy.dotnet.DotnetRunner --master spark://10.151.100.2:7077 --conf DOTNET_WORKER_DIR=K:/Microsoft.Spark.Worker-1.0.0 --conf DOTNET_ASSEMBLY_SEARCH_PATHS=K:/dotnetapp_binaries/Debug/ microsoft-spark-3-0_2.12-1.0.0.jar .\dotnetapp1.exe

it didn't work.

Any help/suggestion on this is appreciated.

[msgurikar]

Thank you David for your inputs.
I am not trying to run on Yarn, i am trying to run in our development physical cluster.
Yes. i have shared Microsoft.Spark.Woker and our code binaries in windows network drive that is accessible to master and worker machines in cluster, then i set DOTNET_WORKER_DIR=network share drive path of Microsoft.Spark.Woker and DOTNET_ASSEMBLY_SEARCH_PATHS=network share drive path of our code binaries.

I am working on this, had some difficulty in accessing network drive.

Thank you.