-
I was excited to see the 2019 example, but it seems to be incomplete. The current microsoft-spark-xxx.jar files throw an error when specifying kafka as format on a readstream. That makes sense to me seeing how Java users must include additional libraries to do the equivalent. Is this example bunk or are kafka streams actually supported? If so, could someone post some more info on configuration? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments
-
FYI I used the supplied example code for counting words https://github.com/dotnet/spark/blob/main/examples/Microsoft.Spark.CSharp.Examples/Sql/Streaming/StructuredKafkaWordCount.cs with the microsoft-spark-3-0-2.12-2.0.0.jar. I see an older thread on kafka streams, but the poster seemed to be using an earlier version. |
Beta Was this translation helpful? Give feedback.
-
According to the Microsoft site Spark 3.0.1 is the latest supported. I used that, but Maven was pointing to a later release of org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1 when running submit. Adding --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1 to the spark-submit fixed this issue. |
Beta Was this translation helpful? Give feedback.
According to the Microsoft site Spark 3.0.1 is the latest supported. I used that, but Maven was pointing to a later release of org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1 when running submit. Adding --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1 to the spark-submit fixed this issue.