Skip to content

Releases: opensearch-project/data-prepper

1.5.2

28 Oct 16:34
24371a2
Compare
Choose a tag to compare

2022-10-27 Version 1.5.2


Security

Maintenance

  • Added GZIP compression integration tests for S3 source (#1577)
  • Updated smoke tests to run against OpenSearch 1.3.6 (#1955)
  • Updated grpcio to 1.50.0 in smoke tests which takes less time to install (#1954)
  • Updated spring to 5.3.23 (#1967)

2.0.1

28 Oct 16:34
4b7b03e
Compare
Choose a tag to compare

2022-10-24 Version 2.0.1


Bug Fixes

  • Sending events to multiple pipelines would result in unexpected behavior because the event was mutated in all pipelines. Duplicate all events sent through the pipeline sink. #1886
  • Support read S3 objects when keys have spaces. #1923.
  • Delete s3:TestEvent objects in the S3 source to avoid those events getting stuck. #1924
  • Correctly parse duration values of 0s or 0ms. #1910

Security

Maintenance

  • Support multiple threads in the Data Prepper core server. #1951
  • Run smoke tests against OpenSearch 1.3.6. #1962
  • Use Python grpcio 1.50.0 in smoke tests to reduce time to run. #1959
  • Reduce test flakiness in RandomStringSourceTests. #1964
  • Reduce smoke test timeout to 8 minutes from 30 minutes. #1965

2.0.0

10 Oct 23:32
0d07a46
Compare
Choose a tag to compare

2022-10-10 Version 2.0.0


Breaking Changes

  • Replaced the data-prepper-tar-install.sh script with bin/data-prepper
  • Replaced the single Jar file with a directory structure and the bin/data-prepper script.
  • Data Prepper pipelines no longer support prepper:. Use processor: instead.
  • Data Prepper now requires Java 11 or higher to run.
  • Removed properties from the opensearch sink: trace_analytics_raw and trace_analytics_service_map. Use the index_type instead.
  • Renamed two grok metric names. Renamed grokProcessingMatchSuccess to grokProcessingMatch and grokProcessingMatchFailure to grokProcessingMismatch.
  • Removed record_type from otel_trace_source.
  • Removed otel_trace_raw_prepper - use otel_trace_raw instead.
  • Removed otel_trace_group_prepper - use otel_trace_group instead.
  • Removed peer-forwarder processor plugin.
  • Incorrect HTTP methods were removed from Data Prepper core HTTP APIs.
  • All APIs have been renamed to use the org.opensearch.dataprepper package.

Features

  • Support conditional routing in sinks (#1337)
  • Support Core Peer Forwarder which replaces the peer forwarder processor plugin (#700)
  • Support parsing CSV and TSV values in event fields using CSV processor (#1081)
  • Support decoding CSV or TSV S3 objects using CSV codec in S3 source (#1644)
  • Support processing JSON values in event fields using JSON processor (#831)
  • Support multi node aggregation using Core Peer Forwarder (#978)
  • Use peer forwarding in service_map_stateful processor using Core Peer Forwarder (#1765)
  • Use peer forwarding in otel_trace_raw processor using Core Peer Forwarder (#1766)

Enhancements

  • Updates default configurations for bounded_buffer - buffer_size is 12,800, batch_size is 200.
  • Support concurrency in Data Prepper expression (#1189)
  • Support bulk create option in OpnSearch sink (#1561)
  • Added new metric to track utilization rate of buffer (#1817)
  • Support publishing metrics for Data Prepper core (#1789)
  • Support configurable timeouts for processors and sinks to flush the data downstream before shutdown (#1742)
  • Moved the fields from S3 JSON objects up to the root level of the object to promote consistency with other S3 codecs (#1687)
  • Support Duration in Data Prepper server configuration (#1623)
  • Fixed health check bug when Auth is enabled for HTTP and OTel trace source (#1600)
  • Enabled HTTP health check for OTel trace source and OTel metrics source (#1546)
  • Made Java 11 to be the baseline version for Data Prepper (#1422)
  • Updated HTTP source request timeout to use configured timeout (#975)
  • Distribute Data Prepper Docker image with JDK 17 (#694)
  • Support ACM and S3 for TLS/SSL in HTTP source (#365)
  • Updated HTTP methods supported on core API endpoints (#313)

Bug Fixes

  • Fix a bug where file sink fails to write to output file with multiple pipeline threads (#1843)
  • Fixed a bug S3 source poll delay (#1550)
  • Fixed a bug where Data Prepper stops if there's an error reading S3 object (#1544)

Infrastructure

  • Improved Data Prepper assemble task to create a runnable distribution (#1762)
  • Updated end-to-end tests with core peer forwarder for trace tests (#1866)
  • Support directory structure for Data Prepper (#305)

Maintenance

  • Removed support for OTLP protocol as internal data transfer in trace pipeline (#1272)
  • Update Jackson to 2.13.4 (#1871)
  • Updated Armeria to 1.19.0 (#1806)
  • Removed Peer Forwarder processor plugin (#1874)
  • Removed deprecated type property in DataPrepperPlugin annotation (#1657)
  • Removed deprecated trace_analytics_raw and trace_analytics_service_map values, index_type replaces them (#1648)
  • Removed deprecated PluginFactory (#1584)
  • Removed AWS SDK v1 entirely from S3 source (#1562)
  • Updated Antlr to 4.10.1 (#1513)
  • Updated all existing prepper plugins to use only processor. (#647)
  • Removed deprecated prepper plugin type from pipeline configuration. (#619)
  • Updated package naming to org.opensearch from com.amazon (#344)

1.5.1

08 Jul 18:00
7e2ad24
Compare
Choose a tag to compare

2022-07-08 Version 1.5.1


Bug Fixes

  • Fix a bug where the S3 Source failed to load all records for concatenated gzip files. This was found in some Application Load Balancer log files. #1572
  • The S3 Source continues to run even if errors occur reading from S3. Previously, the polling thread would stop. #1566
  • S3 Source poll delay will no longer sleep if some messages are received from the SQS queue. #1567

1.5.0

24 Jun 00:45
c4e9e85
Compare
Choose a tag to compare

2022-06-23 Version 1.5.0


Features

  • Support S3 and SQS as a source of events. (#251)
  • Data Prepper can now report its own metrics with custom tags applied. (#1415)
  • Support the Embedded Metrics Format (EMF) for reporting Data Prepper's own metrics. (#1404)

Enhancements

  • The OpenSearch sink now supports disabling any index management from Data Prepper via the management-disabled index-type. (#1051)
  • Add a health check to the HTTP source. (#1466)
  • Display the port number when starting the HTTP source or OTel trace source. (#1469)
  • Allow for HTTP decorators in gRPC authentication plugins to give access to the HTTP request. (#1529)

Bug Fixes

  • Fix a bug where a null plugin setting throws an exception when attempting to validate that setting. (#1525)

Infrastructure

  • Data Prepper supports Docker labels for major version only now. This gets the latest within an entire major version series. (#1475)

Documentation

  • Clarified the latest tested version of OpenDistro in the documentation. (#1494)

Maintenance

  • Updated Armeria to 1.16.0 (#924)
  • Consolidated all AWS SDK usage on v2 (#818)
  • Fix remaining failing unit tests on Windows (#970)
  • Build on Java 11 using the Java 8 toolchain (#1406)
  • Support building on JDK 17 (#1430)
  • Added OpenSearch 2.0.0 to the GitHub Action integration tests (#1467)

1.4.0

17 May 20:59
daf5fc3
Compare
Choose a tag to compare

2022-05-17 Version 1.4.0


Features

Enhancements

  • Migrate Trace Analytics plugins to Event Model (#1216, #1223, #1224, #1220, #1237, #1241, #1239)
  • Support for OpenSearch 2.0 (previously supported versions of OpenSearch and OpenDistro are still supported)

Infrastructure

  • Added needs-documentation label (#1373)
  • Upload and publish JUnit test reports for some tests (#1336)

Documentation

  • Performance tests documentation (#1309)
  • Added custom metrics in otel-trace-source (#1246)

Maintenance

  • Update to use opensearch-java client instead of Rest High Level Client for bulk requests. (#1381)
  • OpenSearch build files clean-up (#1315)
  • Set 30 minute timeout to release process GitHub actions (#1392)
  • Updated Gradle to 7.4.2 (#1377)
  • Fix file uploads to S3 with Gradle 7 (#1383)
  • Updated README links (#1376)
  • Fix link to NOTICE file (#1268)
  • Run OpenSearch sink integration tests against more versions of OpenDistro (#1348)
  • Updated Mockito in OpenSearch plugin (#1339)
  • Removed OpenSearch build-tools gradle plugin from OpenSearch plugin (#1327)
  • Use complete url for processor READMEs (#1324)

Refactoring

  • Made the BulkRetryStrategyTests less reliant on implementation specifics from OpenSearch (#1346)
  • Decouple OpenSearchSinkIT from OpenSearch Core test cases (#1325)
  • Use MatcherAssert.assertThat in OpenSearchSinkIT (#1323)

1.3.0

23 Mar 01:35
f817ef3
Compare
Choose a tag to compare

2022-03-22 Version 1.3.0


Important Deprecation

  • We have updated the pipeline definition to support processor: as replacement of prepper: which has been deprecated and will be fully removed in 2.0. (#655, #667)

Features

  • AggregateProcessor for generic stateful aggregation (README) (#839, #850, #931, #969, #1022, #1046)
  • DateProcessor to extract dates from fields in events (README) (#971, #1014)
  • Processors to support mutate, alter, and delete fields from Events (README) (#1002)
  • KeyValueProcessor to support parsing messages with key-value strings such as queries and properties (README) (#872)
  • DropProcessor to filter out (remove/drop) entire events based off confitional expression (README) (#801, #1174)

Enhancements

  • Add dependency Injection support for Data Prepper Core and Plugins. (#815, #846, #1140)
  • Add Data Prepper expression evaluator (#1024, #1027, #1090, #1065, #1153, #1155, #1157, #1169, #1177, #1178)
  • Support for nested syntax in LogStashConfigConverter (#1088)
  • Support default values for attributes in LogstashConfigConverter mapping files (#1095)
  • Support converting index with a date-time pattern in LogstashConfigConverter (#1045, #1095)
  • Support creation from raw text string in JacksonEvent builder (#770, #1074)
  • Support loading plugins from multiple packages (#948)
  • Support of date and time patterns in opensearch sink index names (#788, #833)
  • Support for passing a PipelineDescription in @DataPrepperPluginConstructor (#825)
  • Validate Plugin Configurations using JSR-303 (#826)
  • Support Logstash configuration conversion for OpenSearch Logstash output (#756)
  • Support negation of boolean attribute values in Logstash configuration converter while mapping plugins (#756)

Bug Fixes

  • Allow stdout and file sink to output generic object type (#1192)
  • Fixed issue where Spring was unable to find the PrometheusMeterRegistry Bean (#1019)
  • Use BlockingTaskExecutor in OtelTraceSource (#745)

Infrastructure

  • Upload Maven artifacts as part of Release build (#1181)
  • Updated the release build to push the Docker image and upload archives (#1151)
  • Update Gradle project to produce only tar.gz archives (#1132)
  • Add simple integration tests for AggregateProcessor (#1046)
  • Assemble the data-prepper-core uber-jar using Zip64 (#820)

Documentation

  • Example on log ingestion from Kubernetes containers (#729)
  • Update copyright headers in release scripts (#933)
  • Update copyright headers in data-prepper subprojects (#776, #928)
  • Example on Data-Prepper ECS Firelens integration (#704)

Maintenance

  • Delete outdated Kibana trace analytics example (#1135)
  • Update Open Distro usages to OpenSearch in scripts (#1086)
  • Upgrade docker-compose.yml files from ODFE to OpenSearch (#847)
  • Update the ADOT example to OpenSearch (#703)
  • Migrate demo sample application to use opensearch and opensearch dashboards instead of ODFE and kibana (#666)
  • Update log ingestion example to use latest data-prepper docker image (#752)

Refactoring

  • Deprecating Record and RecordMetaData (#954)
  • FileSink to use Event model (#750)
  • StringPrepper to use Event model (#753)
  • StdInSource, RandomStringSource to use Event model (#625)

1.2.1

21 Dec 22:04
16fc8fe
Compare
Choose a tag to compare

2021-12-21 Version 1.2.1


Bug Fixes

  • Fix boolean conversion and integer conversion bug for Logstash configuration converter attributes (#766), (#772)

Maintenance

  • Uses Log4j 2.17.0 which fixes CVE-2021-45105 (#764)
  • Updated THIRD-PARTY PARTY file and version to 1.2.1 (#769)

1.2.0

16 Dec 00:42
e3d5b67
Compare
Choose a tag to compare

2021-12-15 Version 1.2.0


Features

Enhancements

  • PluginSettings now supports generic List and Map data types (#302)
  • A disabled SSL warning was added to HTTP and Otel Trace Source plugins and Data Prepper core APIs. A warning will appear in Data Prepper logs when SSL is disabled. (#537) & (#603)
  • HTTP & Otel Trace Source support configurable basic authentication via plugins (#570), (#545)
  • File Source file type is now configurable and supports parsing JSON files. The default remains plain text. (#601),
  • Buffer Interface now supports batch writing via a writeAll method. BlockingBuffer now supports the writeAll method. (#320)
  • Plugin framework now supports a DataPrepperPluginConstructor annotation for indicating a plugin constructor. ($481)
  • Data Prepper core APIs now support basic HTTP Authentication, Docker image's core API's are now secure by default (#558), (#561)
  • OpenSearch sink now supports forwarding requests through an HTTP Proxy (#479
  • OpenSearch sink now supports an optional index-type parameter. (#480)) & (#433)
  • OpenSearch sink now emits a new metric for bulkRequestSizeBytes. (#572)

Bug Fixes

  • Fixed Github Actions for ODFE integration tests (#393)

Infrastructure

  • Using Armeria client builder to help mitigate flaky end-to-end tests (#375)
  • Syncing OpenSearch version to help mitigate flaky end-to-end tests (403)
  • Refactoring existing end-to-end tests out of data prepper core into a new e2e-test module (#512)
  • Added basic grok end-to-end tests and created necessary CI workflow (#536)
  • Code coverage comment bot was added to the GitHub workflow (#549)
  • Added DCO check for GitHub workflow (#360)
  • Code checkstyle integration (#378)
  • Improved Issue template (#397)
  • Supporting Maven publication of the Data Prepper API (#596), (#634) & (#635)
  • Added support for generating THIRD PARTY licenses (#621) & (#631)

Documentation

  • New getting started, developer, getting started trace analytics and pipeline setup guide (#346)
  • Added new log ingestion guide showcasing new HTTP and Grok Prepper Plugins (#573)
  • New guide for migrating to OpenSearch Data Prepper from Open Distro Data Prepper (#470)
  • Added Project Resource to documentation (#482)
  • Added Coding Guidance to the Developer Guid (#560)
  • Added instructions to build and run the Docker image locally (#564)
  • Updated copyright headers for root project, api and core (#569)
  • Update documentation to use OpenSearch Dashboards (#658)
  • Improving OpenSearch sink documentation (#553), (#562), & (#563)

Maintenance

Refactoring

  • Created a new internal data model, Events, to capture data as it flows through the pipeline. This was introduced to eliminate the excessive de/serialization of the current implementation. Currently, integrated with only log ingestion and sample plugins (#), (#435), (#463), (#468), (#477), & (#539)
  • The StdOutSink supports Objects instead of Strings as part of the migration to support the new event model (#599)
  • The FileSource uses Objects to support the new event model (#)
  • Small refactoring of PeerForwarder to improve readability of the code (#626)
  • Plugin class redesign leveraging new plugin framework (#363), (#451), (#478)
  • Consistent usage of OpenSearch, OpenSearch Dashboards and Amazon OpenSearch. (#637)
  • Ref...
Read more

1.1.1

10 Dec 23:22
204d0f9
Compare
Choose a tag to compare

Data Prepper 1.1.1
Release Notes