Release 1.2.0 · opensearch-project/data-prepper

2021-12-15 Version 1.2.0

Grok Prepper for processing unstructured data with grok pattern matching. (#302), (#324), (#377), (#449), (#510), (#548), (549), (#548), & (#586)
HTTP Source plugin for receiving log data (#309), (#325), (#359), (#380), & (#415)
Logstash config support. Users can now run Data Prepper with a logstash.conf file. (#581), (#568), (#579), (#580), (#582), (#587), (#636), (#473), (#535), (#552), (#577), (#588), (#597), (#616), (#447), (#506), (#559), (#575), (#584), (#591), & (#617)

PluginSettings now supports generic List and Map data types (#302)
A disabled SSL warning was added to HTTP and Otel Trace Source plugins and Data Prepper core APIs. A warning will appear in Data Prepper logs when SSL is disabled. (#537) & (#603)
HTTP & Otel Trace Source support configurable basic authentication via plugins (#570), (#545)
File Source file type is now configurable and supports parsing JSON files. The default remains plain text. (#601),
Buffer Interface now supports batch writing via a writeAll method. BlockingBuffer now supports the writeAll method. (#320)
Plugin framework now supports a DataPrepperPluginConstructor annotation for indicating a plugin constructor. ($481)
Data Prepper core APIs now support basic HTTP Authentication, Docker image's core API's are now secure by default (#558), (#561)
OpenSearch sink now supports forwarding requests through an HTTP Proxy (#479
OpenSearch sink now supports an optional index-type parameter. (#480)) & (#433)
OpenSearch sink now emits a new metric for bulkRequestSizeBytes. (#572)

Using Armeria client builder to help mitigate flaky end-to-end tests (#375)
Syncing OpenSearch version to help mitigate flaky end-to-end tests (403)
Refactoring existing end-to-end tests out of data prepper core into a new e2e-test module (#512)
Added basic grok end-to-end tests and created necessary CI workflow (#536)
Code coverage comment bot was added to the GitHub workflow (#549)
Added DCO check for GitHub workflow (#360)
Code checkstyle integration (#378)
Improved Issue template (#397)
Supporting Maven publication of the Data Prepper API (#596), (#634) & (#635)
Added support for generating THIRD PARTY licenses (#621) & (#631)

New getting started, developer, getting started trace analytics and pipeline setup guide (#346)
Added new log ingestion guide showcasing new HTTP and Grok Prepper Plugins (#573)
New guide for migrating to OpenSearch Data Prepper from Open Distro Data Prepper (#470)
Added Project Resource to documentation (#482)
Added Coding Guidance to the Developer Guid (#560)
Added instructions to build and run the Docker image locally (#564)
Updated copyright headers for root project, api and core (#569)
Update documentation to use OpenSearch Dashboards (#658)
Improving OpenSearch sink documentation (#553), (#562), & (#563)

Updated version to 1.2 (#416)
The OpenSearch REST client and the build plugins used by the OpenSearch plugin are now at 1.1.0 (#384)
Use OpenSearch instead of Elasticsearch in builds (#438)
Use Netty 4.1.68 which fixes CVE-2021-37136 and CVE-2021-37137 (#661)
Uses Log4j 2.16.0 which fixes CVE-2021-44228 and CVE-2021-45046 (#742)

Created a new internal data model, Events, to capture data as it flows through the pipeline. This was introduced to eliminate the excessive de/serialization of the current implementation. Currently, integrated with only log ingestion and sample plugins (#), (#435), (#463), (#468), (#477), & (#539)
The StdOutSink supports Objects instead of Strings as part of the migration to support the new event model (#599)
The FileSource uses Objects to support the new event model (#)
Small refactoring of PeerForwarder to improve readability of the code (#626)
Plugin class redesign leveraging new plugin framework (#363), (#451), (#478)
Consistent usage of OpenSearch, OpenSearch Dashboards and Amazon OpenSearch. (#637)
Refactoring OpenSearch sink to improve extendability and maintainability. (#361) & (#414)
Performance Improvements for Event model and Prepper threading (#728), (#708)