Release 2.6.0 · opensearch-project/data-prepper

2023-11-28 Version 2.6.0

Support DynamoDB as a source. (#2932)
Use Kafka as a buffer (#3322)
Support dynamically changing the visibility timeout for S3 Source with SQS queue (#2485)
Create or update Amazon OpenSearch Serverless network policy (#3577)
Sink level metric for end to end latency (#3494)

Use Amazon Linux as base Docker image (#3505)
Allow the Kafka buffer (and others that do not require the heap) to bypass the heap circuit breaker (#3616)
Improve gRPC request exception logging (#3621)
Configure the delay in the random string source (#3601)
Add distribution_version flag to opensearch source (#3636)

Data Prepper is writing empty DLQ objects (#3644)
Bulk Operation Retry Strategy should print cause of error (#3504)
ISM index rollover actions fail because of missing setting for otel-v1-apm-span-* indices (#3506)
AWS opensearch source error: ElasticsearchVersionInfo.buildFlavor (#3640)
No permissions for writing to Amazon OpenSearch Serverless collection only shows errors after max_retries limit is reached (#3508)
Bulk Operation Retry Strategy should print cause of error (#3504)
NullPointer exception in DefaultKafkaClusterConfigSupplier get API (#3528)
Fix bug so global read-only items do not expire from TTL in DynamoDB source coordination store (#3703)
Check if failedDeleteCount is positive before logging an SQS error (#3686)
Docker image jre-jammy contains Berkeley DB (#3543)
Race condition in DefaultEventHandle (#3617)

CVE-2023-44981 (Critical) detected in multiple libraries (#3491)
CVE-2023-36478 (High) detected in http2-hpack-11.0.12.jar, jetty-http-11.0.12.jar (#3490)
CVE-2023-4586 (High) detected in netty-handler-4.1.100.Final.jar (#3443)
CVE-2023-5072 (High) detected in json-20230618.jar (#3522)
CVE-2023-39410 (High) detected in avro-1.11.0.jar (#3430)
CVE-2023-4043 (High) detected in parsson-1.1.2.jar (#3588)
CVE-2023-46122 (High) detected in io_2.13-1.9.1.jar (#3547)
CVE-2023-46136 (High) detected in Werkzeug-2.2.3-py3-none-any.whl (#3552)
CVE-2023-26048 (Medium) detected in jetty-server-11.0.12.jar (#2533)
CVE-2023-26049 (Medium) detected in jetty-http-11.0.12.jar, jetty-server-11.0.12.jar (#2532)
CVE-2023-40167 (Medium) detected in jetty-http-11.0.12.jar (#3359)
CVE-2023-36479 (Medium) detected in jetty-servlets-11.0.12.jar (#3367)
WS-2023-0236 (Low) detected in jetty-xml-11.0.12.jar (#3072)

Update to the Gradle 8.x version which supports Java 21. Gradle 8.3 is supporting up to Java 20. (#3330)
Start building Data Prepper on Java 21 (#3329)
Integration tests to validate data going to OpenSearch (#3678)
Unit tests fail on Windows machine (#3459)
Fix disabled E2E ack integration tests in PipelinesWithAcksIT.java (#3472)
Remove the @Deprecated from Record (#3536)
Remove all unnecessary projects in the 2.6 branch (#3605)
Update end-to-end tests to run from the released Docker image (#3566)