Microsoft.Spark.Worker longevity

[dbeavon]

I'm getting started with an attempt to convert some of our data warehousing to .Net for Spark. It has been a fairly easy migration so far (from Scala). As of now I've only been working on it for a couple days. But I am much more productive in .Net. It takes me only a few hours to do stuff that used to take days in Scala. And not only do we get to use .Net - but I get to use VS too (instead of IntelliJ/modules and databricks/notebooks). Thanks to all of you who worked on this!

My question is related to the underlying architecture. I was wondering about the worker processes that are spawned. Why do these processes (Microsoft.Spark.Worker) keep stopping and restarting?

Can't the Spark environment keep things running along side the worker nodes without killing the .Net processes all the time? It seems like there would be some OS overhead to continually restart these processes and reload assemblies. It doesn't seem optimal. Does the python architecture do a similar thing?

[imback82]

Are you running this on Windows? If so, worker needs to be respawned for each task: https://github.com/apache/spark/blob/2fa792aa64f6153af1641d895e2f996b18dfbce4/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala#L48

On non-Windows, one worker is launched per executor and shared across tasks.

Note that we are piggybacking on how Python worker is invoked, there is not much we can do to improve this process unless we modify the OSS Spark.

[dbeavon]

@imback82
Thanks for the reply.

Yes on Windows. I was just a bit surprised to see it. Sometimes Spark seems like it was developed to be fast, and other times it doesn't seem that way. I'm guessing the overhead of restarting processes is not a huge factor in the grand scheme of things. But its sad to see that they are taking shortcuts.

I see they relied on signals to make things work as desired in UNIX.

IMHO those POSIX signals seem so limiting and arbitrary to me, even after working on HP-UX for 20 years. It seems almost as silly as building an IPC system for Windows that is based on a predefined set of WIN32 messages for the windows message pump. Oh well, maybe after another 20 years I'll start to appreciate signals a bit more.

[imback82]

I'm guessing the overhead of restarting processes is not a huge factor in the grand scheme of things.

That's correct. If your task is big (e.g., applying UDFs to millions of rows), the cost of starting a new worker process becomes negligible anyway. I actually benchmarked the difference on Linux (which supports both mode) with TPC-H (1000SF) in 2019, and didn't find much difference.

[dbeavon]

@imback82 You mentioned that if the task is big, then the overhead of starting processes is negligible. That may be true in production but when we are doing development work, we don't have millions of rows. We generally only have a few thousands. I am mostly just concerned about getting my work done without so getting bottlenecked on CPU all day long. I don't care how my stuff will perform in production. That is someone else's problem. ;-)

FYI, What I've been struggling with the most is not CPU consumed directly by Spark or the Microsoft.Spark.Worker , but it is the CPU consumed by antivirus (mssense and msmpeng). The antivirus on my workstation is not happy about the rapid rate at which we are launching processes. The antivirus is probably consuming more CPU than all of the work being done by Spark!

Unfortunately in some larger companies, a developer isn't able to adjust their own exclusion rules. Hopefully I will get a chance to work with another internal team next week, and reduce the amount of interference from the antivirus software.

[imback82]

If this is for development work, can you have the minimum number of executor for your job to reduce the frequency of launching the worker processes?

[dbeavon]

@imback I only had two executor processes in my standalone cluster.

I finally figured out what was going on. It was very odd behavior. Even though I had only 1 row in a dataframe, there were dozens of Microsoft.Spark.Workers being launched for the sake of running the UDF. After investigating the Spark U/I, it showed 200 tasks being used for a single row. I ended up using Coalesce() on the dataframe before starting the UDF operation. That seems to have resolved the problem with the rapid spawning of .Net processes, as well as the large amount of antivirus CPU.

I'm really not clear on how I ended up with so many partitions.... Previously I had been using calls to Persist(MEMORY_ONLY) and Checkpoint() for my local development. This was primarily to avoid retrieving data from ADLS repeatedly. But I guess I will have to remember this Coalesce() trick now too. Perhaps this issue is only applicable to Windows where these processes are restarted so frequently?