Stream Reactor 7.4.5

7.4.4 - Add SSL support to Elastic and fix of NPE in cloud storage sinks

Stream Reactor 7.4.4

Azure Datalake & GCP Storage

Dependency version upgrades

Data Lake Sinks (AWS, Azure Datalake and GCP Storage)

• Fixes a gap in the avro/parquet storage where enums where converted from Connect enums to string.
• Adds support for explicit "no partition" specification to kcql, to enable topics to be written in the bucket and prefix without partitioning the data.
  • Syntax Example: INSERT INTO foo SELECT * FROM bar NOPARTITION

Stream Reactor 7.4.3

All Connectors

Dependency version upgrades

Data Lake Sinks (AWS, Azure Datalake and GCP Storage)

This release introduces a new configuration option for three Kafka Connect Sink Connectors—S3, Data Lake, and GCP Storage—allowing users to disable exactly-once semantics. By default, exactly once is enabled, but with this update, users can choose to disable it, opting instead for Kafka Connect’s native at-least-once offset management.

Configuration Parameters:

S3 Sink Connector: connect.s3.exactly.once.enable
Data Lake Sink Connector: connect.datalake.exactly.once.enable
GCP Storage Sink Connector: connect.gcpstorage.exactly.once.enable

Default Value: true

Stream Reactor 7.4.2

Releasing 7.4.2 with aws s3 source extension exclusion filters, updat…

Stream Reactor 7.4.1

Tagging for 7.4.1

Stream Reactor 7.4.0

Fix: Java class class java.util.Date   (#58)

* Fix: Java class class java.util.Date does not have corresponding schema type.\n\tat org.apache.kafka.connect.json.JsonConverter.convertToJson

When the payload handed over to the sink in the form of a java.util.Map[_, _], and one value in the Map is of java.util.Date the conenctor fails with the exception above.

To fix the issue, the code relies on Jackson to serialise the code.

* Support the Map with Struct as part of the JSON conversion.

---------

Co-authored-by: stheppi <[email protected]>

Stream Reactor 7.3.2

Release 7.3.2 - adding date support to cloud connectors, version upgr…

Stream Reactor 7.3.1

GCP Storage - Invalid Protocol Configuration Fix

Stream Reactor 7.3.0

NEW: Azure Service Bus Source Connector

• Azure Service Bus Source Connector Find out more.

NEW: GCP Pub/Sub Source Connector

• GCP Pub/Sub Source Connector Find out more.

Data Lake Sinks (AWS, Azure Datalake and GCP Storage)

• To back the topics up the KCQL statement is

INSERT INTO bucket
SELECT * FROM `*`
...

When * is used the envelope setting is ignored.

This change allows for the * to be taken into account as a default if the given message topic is not found.

HTTP Sink

Bug fix to ensure that, if specified as part of the template, the Content-Type header is correctly populated.

All Connectors:

• Update of dependencies.

Note

Upgrading from any version prior to 7.0.0, please see the release and upgrade notes for 7.0.0.

Stream Reactor 7.2.0

Enhancements

1. Automated Skip for Archived Objects:

   • The S3 source now seamlessly bypasses archived objects, including those stored in Glacier and Deep Archive. This enhancement improves efficiency by automatically excluding archived data from processing, avoiding the connector crashing otherwise

2. Enhanced Key Storage in Envelope Mode:

   • Changes have been implemented to the stored key when using envelope mode. These modifications lay the groundwork for future functionality, enabling seamless replay of Kafka data stored in data lakes (S3, GCS, Azure Data Lake) from any specified point in time.

Full Changelog: 7.1.0...7.2.0