[CT-617] [Spark]: Upgrade the version of spark to 3.x

[nssalian]

Describe the feature

The spark test image is built with a Spark 2 tag, while the open-source Spark is further into Spark 3.x.
This issue focuses on the upgrade to Spark 3 - it would unlock a host of new features that come from the migration from 2 to 3.
Proposing to use Spark 3.1.1

Concern/ future items

We should evaluate if the godatadriven/spark image is the best way forward for the community. It's versions don't seem to be kept up-to-date with the open source version. This doesn't have to occur in this issue but we should look at it in the future.

Are you interested in contributing this feature?

I'd be interested to open a PR to focus on the upgrade.

[TalkWIthKeyboard]

@nssalian Hi, I am upgrading the Spark version too, we meet the same problem that tests get stuck for a long time, I suspect it may be caused by the resource size of the test container, if there are any other clues, I would be interesting to discuss the details of the problem with you and work it out.

[nssalian]

Thanks, @TalkWIthKeyboard. I bumped up the size and it didn't help. I'm thinking it might be related to Spark and Thrift itself. I'll try more debugging tomorrow. Once we merge #349, I can help you get #342 functionality in.

[TalkWIthKeyboard]

@nssalian Thanks, looking forward to your work.

[leahwicz]

@nssalian #349 completed this work, correct? I'm closing this out but if there is still more to do, just let me know