support Spark 3.3.1 version

[pawel-big-lebowski]

Signed-off-by: Pawel Leszczynski [email protected]

Problem

Include support for the latest Spark 3.3.1 version.

Solution

Update Spark version and fix issue introduced by upgraded Jackson within Spark.

Note: All schema changes require discussion. Please link the issue for context.

• Your change modifies the core OpenLineage model
• Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

Checklist

• You've signed-off your work
• Your pull request title follows our guidelines
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
• You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
• You've added a header to source files (if relevant)

[pawel-big-lebowski]

looks like a Jackson upgrade occurred:
https://issues.apache.org/jira/browse/SPARK-40782

[mobuchowski]

@pawel-big-lebowski another tree needed? 😉

[pawel-big-lebowski]

@pawel-big-lebowski another tree needed? 😉

Hopefully not. This look like some Spark & Gradle issue that was already merged to Spark master:
FasterXML/jackson-bom#52 (comment)
apache/spark#38355

[pawel-big-lebowski]

Need to wait for docker image bitnami/spark:3.3.1 to proceed.