Skip to content

2.0.0
Compare
Choose a tag to compare
@dlvenable dlvenable released this 10 Oct 23:32
· 1527 commits to main since this release
2.0.0
0d07a46
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

2022-10-10 Version 2.0.0

Breaking Changes

  • Replaced the data-prepper-tar-install.sh script with bin/data-prepper
  • Replaced the single Jar file with a directory structure and the bin/data-prepper script.
  • Data Prepper pipelines no longer support prepper:. Use processor: instead.
  • Data Prepper now requires Java 11 or higher to run.
  • Removed properties from the opensearch sink: trace_analytics_raw and trace_analytics_service_map. Use the index_type instead.
  • Renamed two grok metric names. Renamed grokProcessingMatchSuccess to grokProcessingMatch and grokProcessingMatchFailure to grokProcessingMismatch.
  • Removed record_type from otel_trace_source.
  • Removed otel_trace_raw_prepper - use otel_trace_raw instead.
  • Removed otel_trace_group_prepper - use otel_trace_group instead.
  • Removed peer-forwarder processor plugin.
  • Incorrect HTTP methods were removed from Data Prepper core HTTP APIs.
  • All APIs have been renamed to use the org.opensearch.dataprepper package.

Features

  • Support conditional routing in sinks (#1337)
  • Support Core Peer Forwarder which replaces the peer forwarder processor plugin (#700)
  • Support parsing CSV and TSV values in event fields using CSV processor (#1081)
  • Support decoding CSV or TSV S3 objects using CSV codec in S3 source (#1644)
  • Support processing JSON values in event fields using JSON processor (#831)
  • Support multi node aggregation using Core Peer Forwarder (#978)
  • Use peer forwarding in service_map_stateful processor using Core Peer Forwarder (#1765)
  • Use peer forwarding in otel_trace_raw processor using Core Peer Forwarder (#1766)

Enhancements

  • Updates default configurations for bounded_buffer - buffer_size is 12,800, batch_size is 200.
  • Support concurrency in Data Prepper expression (#1189)
  • Support bulk create option in OpnSearch sink (#1561)
  • Added new metric to track utilization rate of buffer (#1817)
  • Support publishing metrics for Data Prepper core (#1789)
  • Support configurable timeouts for processors and sinks to flush the data downstream before shutdown (#1742)
  • Moved the fields from S3 JSON objects up to the root level of the object to promote consistency with other S3 codecs (#1687)
  • Support Duration in Data Prepper server configuration (#1623)
  • Fixed health check bug when Auth is enabled for HTTP and OTel trace source (#1600)
  • Enabled HTTP health check for OTel trace source and OTel metrics source (#1546)
  • Made Java 11 to be the baseline version for Data Prepper (#1422)
  • Updated HTTP source request timeout to use configured timeout (#975)
  • Distribute Data Prepper Docker image with JDK 17 (#694)
  • Support ACM and S3 for TLS/SSL in HTTP source (#365)
  • Updated HTTP methods supported on core API endpoints (#313)

Bug Fixes

  • Fix a bug where file sink fails to write to output file with multiple pipeline threads (#1843)
  • Fixed a bug S3 source poll delay (#1550)
  • Fixed a bug where Data Prepper stops if there's an error reading S3 object (#1544)

Infrastructure

  • Improved Data Prepper assemble task to create a runnable distribution (#1762)
  • Updated end-to-end tests with core peer forwarder for trace tests (#1866)
  • Support directory structure for Data Prepper (#305)

Maintenance

  • Removed support for OTLP protocol as internal data transfer in trace pipeline (#1272)
  • Update Jackson to 2.13.4 (#1871)
  • Updated Armeria to 1.19.0 (#1806)
  • Removed Peer Forwarder processor plugin (#1874)
  • Removed deprecated type property in DataPrepperPlugin annotation (#1657)
  • Removed deprecated trace_analytics_raw and trace_analytics_service_map values, index_type replaces them (#1648)
  • Removed deprecated PluginFactory (#1584)
  • Removed AWS SDK v1 entirely from S3 source (#1562)
  • Updated Antlr to 4.10.1 (#1513)
  • Updated all existing prepper plugins to use only processor. (#647)
  • Removed deprecated prepper plugin type from pipeline configuration. (#619)
  • Updated package naming to org.opensearch from com.amazon (#344)
Assets 2
Loading