Synapse Analytics is killing their built-in language support for .Net in the Synapse runtime v.3.3 (and beyond)

[dbeavon]

I had asked for someone in the Synapse organization to share this news with everyone first-hand, but they would not. Please feel free to contact your account manager or Microsoft support (CSS) if you are an Azure customer, and you need more clarification about anything mentioned below. I would be quite happy if I was mistaken about any of this.

Synapse is killing the .Net-for-spark language bindings in the next version of their Spark runtime (v.3.3). They claim it is because the community project only works on the .Net Core 3.1 runtime (end-of-life). They don't seem to realize that this project now offers .Net 6 compatibility as well. Here is the explanation which was shared in their Spark 3.3 release notes:

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-33-runtime

After seeing this, I quickly explained to this Synapse team that the PR to convert to .Net 6 has already been merged into the “main” branch as-of this past week. But I don't think anyone is convinced. In any case, their explanation about the reason for removing .Net is probably just a pretext. Below is a more likely explanation from the team (although still vague). They say they are killing .Net because of "several factors like usage, development work and other priority items."
Please see the email below ...

...

This news seems a bit discouraging at first. And I think it is a setback for the Synapse Analytics. But it is NOT necessarily a setback for this project. Their reasons for killing .Net are primarily based on usage & monetization (and are likely based on losing some engineers as well.) These things are very much a concern for a Microsoft Azure PM. However they may not be as much of a concern to the members of this community. In other words their reasons for changing the roadmap of Synapse has nothing whatsoever to do with either the quality or usefulness of .Net for Spark. (Synapse is not the ONLY platform that is available for hosting Spark workloads.)

There may be more unspoken reasons for their decision as well. My suspicion is that a small team of Synapse engineers has left Microsoft recently, and these team-members were the full-time maintainers of this project. They were probably supposed to assist with .Net customer support in Synapse, to some degree, and a large void was created when they left. They were probably paid from the budgets of Azure (eg. from the Synapse and/or HDI platforms). It seems likely to me that they have left Microsoft to continue the same type of work elsewhere. As a result of these departures, Microsoft probably does not have the bandwidth to support the .Net language bindings in Synapse anymore (probably cannot even do the minimal work to redistribute any deliverables from this project). Everything in this paragraph is conjecture, and is only based on some observations I made here in the community and from what I see in a couple of Linkedin profiles.

I'm only sharing this information because I watched everything happen first-hand. I wanted to make sure that any affected community members have some idea of what happened. Over the course of just a few months the Synapse organization rapidly removed .Net from its long-term roadmap. They did this part-way thru a single support case of mine (a case related to DNS/IPv6). Just a few short months ago, prior to my support case, these C# language bindings were being promoted and were given full support (at least verbally). Based on everything I was told, there was a clear path for .Net programmers to start hosting our solutions on Synapse. This message came from Microsoft PM's, and from Azure pre-sales, and from cloud solutions architects, and from Microsoft support engineers. These .Net bindings in Synapse were promoted as a mature (GA) feature of a mature platform. But based on what I'm hearing now, the .Net bindings in Synapse will only be supported for a little while longer (until the end-of-life for Synapse 3.2. See: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/runtime-for-apache-spark-lifecycle-and-supportability
)

I tried to ask if there may be some flexibility to run .Net extensions without formal support (similar to how things work on every other Spark platform). However I don't think there are any plans for this yet either. I think the team is unlikely to enable this, and it would be difficult to make it work on our own since there is such a small surface area to perform bootstrapping, or configure the cluster nodes.

Given the shortened lifecycle of the .Net language bindings in Synapse, any of you folks who are hosting on Synapse or planning to host on Synapse may need to find another Spark platform. Personally I've already had some good experiences hosting on-premise, on Azure Databricks, and even on Synapse Analytics. I hope to test HD Insight in the next week or so as well. Let me know if I can try to answer any follow-up questions. Sorry for this huge wall of text.

[dbeavon]

Another update. The synapse team is putting this warning into some other parts of their documentation as well:

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/spark-dotnet