3.8.0

This version comes with a new Enrich app, enrich-nsq. Also, it has following improvements:

• Superseding schemas
• Improvements in API/SQL enrichments
• Making derived contexts accesible to the JavaScript enrichment

Superseding schemas

Schemas define the structure of the data that you collect. Each schema defines what fields are recorded with each event that is captured, and provides validation criteria for each field. Schemas are also used to describe the structure of entities that are attached to events.

However, there are some cases where we want to replace schema versions in incoming events with another version due to some problem in the tracking code. The new superseding schemas feature makes this possible.

So, how does this work exactly? If we want a schema to be replaced by another one, we state this with $supersededBy field of the schema. Later, when an event with superseded schema arrived, superseded schema version will be replaced by the specified superseding schema version.

Improvements in API/SQL enrichments

The API enrichment lets you perform dimension widening on a Snowplow event via your own or third-party proprietary http(s) API. The SQL enrichment is the relational database counterpart of the API Enrichment. It allows you to use relational database to perform dimension widening.

Enrich caches the results of API requests and SQL queries with the API/SQL enrichments to avoid continuous calls. We've made some improvements in caching the errors. These improvements are:

• Set TTL for errors to the tenth of TTL for successful results. With this way, API/SQL requests can be retried faster in case of cached error.
• When we get an error, the error will be cached but we will return last known 'old' good value for further processing. This fallback would allow Enrich to produce fewer bad rows in case of 'getting stuck' with errors in the enrichment cache.

More details about the caching improvements can be found here.

Also, we've made some changes to the way we handle database connections with SQL enrichment. These changes should lead to acquiring database connections in a better way and better usage of existing database connections.

enrich-nsq, new member of 2nd generation enrich apps

In this release, enrich-nsq becomes the newest member of 2nd generation Enrich apps. It allows to read from and write to NSQ topics.

Instructions to setup and configure Enrich can be found on our docs website.

Making derived contexts accesible to the JavaScript enrichment

Previously, the JavaScript enrichment allowed users to call event.getDerived_contexts(), however, it was returning always null. Starting with Enrich 3.8.0, it will be possible to access derived contexts in the JavaScript enrichment.

Changelog

• Take superseding schema into account during validation (#751)
• common: Provide derived contexts to JS enrichment (#769)
• Scan Docker images in Snyk Github action (#772)
• common: do not validate enrichment names (#767)
• common: SQL enrichment: get connection only if request not cached (#765)
• common: SQL enrichment: put getConnection in Blocker (#763)
• common-fs2: fix env var substitution for JSON files (#753)
• Add enrich-nsq (#740)
• fix: add mskAuth to kafka depedencies (#746)
• common: improve caching in API/SQL enrichments (#747)

3.7.3

Enrich now ignores (during validation) HTTP resources defined in $ref keyword in Iglu schemas, avoiding expensive external lookups.

Changelog

• enrich-pubsub: check max message attricutes length (#782)
• common: upgrade iglu-scala-client to 1.3.1 (#781)

3.7.2

This release adds ability to decode base64-encoded events if initial Thrift serialization fails.

Changelog

• Try to base64 decode event if initial Thrift serialization fails (#777)

3.7.1

A patch release that bumps dependencies for potential security vulnerabilities.

Changelog

• Bump net.minidev json-smart to 2.4.10 (#771)

3.7.0

Starting from this version, it is now possible to use environment variables in all the config files of the app (application hocon, resolver, enrichments).

It also bumps YAUAA to the latest version, so that it correctly detects Chrome 109.

Last but not least, SQL, API and JS enrichment are now activated in the integration tests on Github Actions and we've added a cache for UA parser enrichment that will improve its performance.

Changelog

• common: bump http4s to 0.21.34 (#744)
• common: bump postgresql to 42.5.2 (#743)
• common: bump netty to 4.1.87.Final (#742)
• common: bump yauaa to 7.11.0 (#741)
• enrich-kinesis: activate JS enrichment, SQL enrichment and API request enrichment in integration tests (#724)
• common: parse Iglu resolver and enrichment configs as HOCON instead of JSON (#736)
• Add UA parser enrichment cache (#738)

Version 3.6.1

Enrich 3.6.1 updates the YAUAA to 7.9.0 because of CVE-2022-23496. Also, it bumps Iglu Scala Client to 1.3.0.

Full changelog

• common: bump iglu scala client to 1.3.0 (#734)
• common: Update Yauaa to 7.9.0 (#734)

Version 3.6.0

Enrich 3.6.0 updates the YAUAA enrichment to support Client Hints and improves the Sendgrid adapter to remove duplicate events from a payload.

Full changelog

• common: Prune system libs from distroless image (#728)
• common: remove duplicates from Sendgrid adapter payload (#725)
• common: Upgrade yauaa to 7.x (#639)

Version 3.5.1

This is a maintenance release, to upgrade dependencies to newer versions

Changelog

• enrich-kinesis: exclude aws glue libs from the build (#722)
• common: bump slf4j to 2.0.3 (#721)
• enrich-pubsub: bump GCP sdk to 2.14.0 (#719)
• common: bump netty to 4.1.84.Final (#718)
• common: bump protobuf-java to 3.21.9 (#717)
• common: bump jackson-databind to 2.13.4.2 (#716)
• common: bump commons-text to 1.10.0 (#715)
• enrich-kinsis: bump amazon-kinesis-client to 2.4.3 (#714)
• enrich-kinesis: exclude amazon-kinesis-producer from build (#713)
• enrich-kinesis: add integration tests with localstack and testcontainers (#706)
• common: use correct event id for PII event's parent context (#709)
• common: bump specs2 to 4.17.0 (#707)
• common: remove common info from v_etl (#704)

3.5.0

This release introduces enrich-kafka, the newest member of 2nd generation enrich apps, following enrich-kinesis and enrich-pubsub!

enrich-kafka is cloud agnostic which means that it can run on AWS, on GCP, on Azure, on-premise or on localhost. Both enrich-kafka and recently announced enrich-rabbitmq-experimental enable us to run Snowplow pipeline anywhere.

Instructions to setup and configure enrich can be found on our docs website.

Version 3.4.1

This release comes with a few improvements.

• Enrich uses latest version of Iglu Scala client that comes with better caching, which improves the performance of enrich.
• enrich-kinesis retries ProvisionedThroughputExceededException errors forever (more details on the issue).
• enrich-kinesis adds STS support, making it possible for Enrich deployed on an EKS cluster to assume irsa role with corresponding permissions.
• Enrich does not try to URL decode the user agent any more. This could lead to bad rows in case the user agent was containing some unallowed characters.

Changelog

• Enrich-kinesis: improve sink retry policy for when throttled by kinesis (#697)
• common: use iglu-scala-client with improved caching (#699)
• enrich-kinesis: add STS v2 to runtime dependencies (#643)
• Common: remove url decoding the user agent (#695)
• enrich-rabbitmq: fix jar patch in Github worflow for releasing (#702)

How to upgrade

If you are already using a recent version for Enrich, then upgrading is as simple as pulling the latest docker image:

docker pull snowplow/snowplow-enrich-pubsub:3.4.1
docker pull snowplow/snowplow-enrich-kinesis:3.4.1
docker pull snowplow/stream-enrich-kafka:3.4.1
docker pull snowplow/snowplow-enrich-rabbitmq-experimental:3.4.1

More information can be found on our docs website.