Atum v3.10.0

Release notes

A new feature #156 has been added in #157 implemented by @Zejnilovic.

Features

• #156 Add support for sum of truncated values

Atum v3.9.0

Release notes

Version of Jackson SerDe has been updated (vulnerability fix):

• For Spark 3: currently latest version 2.14.1
• For Spark 2: 2.12.7 (databind 2.12.7.1)

Issues fixed

#150 Vulnerability fix for jackson databind

Atum v3.8.3

Release notes

• Version of Jackson SerDe has been updated to 2.12.3 for Spark 3.2 (unchanged for Spark 2.4)
• buggy behavior of Accumulator.setAdditionalInfo was fixed

Issues fixed

• #137 Release 3.8.2 broke Spark 3 compatibility
• #139 Cannot add new additionalInfo in metadata

Atum v3.8.2

Release notes

• RunState enums Jackson JSON serialization support was added - backport of #83 already present in v4
• This is a release-fix of v3.8.1 (v3.8.1 is identical to v3.8.0)

Issues fixed

• #131 Backport of RunState enums serialization (#83)

Atum v3.8.1

Due to an error, this version is identical to v3.8.0. Follow the next release for feature updates.

Atum v3.8.0

Release notes

• Atum packages for Scala 2.12 (atum_2.12, atum_model_2.12, and atum-s3-sdk-extension_2.12) are now built against Spark 3.2.1.
• Scala 2.11 version remains the same as in v3.7.0 (built for Spark 2.4)

Issues fixed

• #127 Update to Spark 3.2.1

Atum v3.7.0

Release notes

• The file permissions on HDFS which the _INFO file will be created with are now driven by (implemented in #114):
  • typesafe config key atum.hdfs.info.file.permissions (e.g. set atum.hdfs.info.file.permissions=711)
  • default HDFS FsPermissions with umask applied

Please note, that the target umask will work as a cap for your desired permissions (e.g. having umask=022, your desired 777 permissions will become 755)

Issues fixed

• #113 Making _INFO file permission configurable

Atum v3.6.0

Release notes

• When creating a ControlMeasure from scratch using the ControlMeasureBuilder, the user can now specify the controlTypes for specific columns to be used. Documentation in the README added.

Issues fixed

• #97 Enable user to specify measurement controlType

Atum v3.5.1

Release notes

• Ability to prepend a checkpoint to control measure has been added. Documentation in the README added.

Issues fixed

• #75 Ability to prepend a checkpoint

Atum v3.5.0

Release notes

• Improvement: implicit SparkSessionWrapper renamed to AtumSparkSessionWrapper in order to prevent name clashing.
• Generalization: S3Location Utils moved to AbsaOSS/commons, so a migration for S3Location-related routines is needed:
  • needs dependency za.co.absa.commons:commons_2.11:0.0.27 (for Scala 2.12 use commons_2.12)
  • za.co.absa.atum.location.S3Location -> za.co.absa.commons.s3.S3Location
  • za.co.absa.atum.location.S3Location.StringS3LocationExt -> za.co.absa.commons.s3.SimpleS3Location.SimpleS3LocationExt
  • address small API differences:
    • "string".toS3Location -> "string".toSimpleS3Location
    • s3Location.s3String -> s3Location.asSimpleS3LocationString
• Added support for Apache Spark 3.1.X
• Atum model now has json4s dependencies as provided dependencies to accommodate multiple Spark versions. More details in the README.

Issues fixed

• #69 Atum implicit rename
• #71 S3Location moved to AbsaOSS/commons