From 3017f515b7c2a2173b0fe764f29805ab27f63f47 Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Mon, 7 Nov 2022 09:25:47 -0700 Subject: [PATCH] Logstash migration guide (#1487) * Added cluster permissions to cluster permissions list. Signed-off-by: carolxob * Revert "Added cluster permissions to cluster permissions list." This reverts commit 48a9fb56b7fce308eccf2b4dd8ad6b7c96515a4d. * Migrated pipelines and migrating from Open Distro. Signed-off-by: carolxob * Migrated Logstash Migration Guide to doc site. Signed-off-by: carolxob * Update _clients/data-prepper/migrate-open-distro.md Co-authored-by: Heather Halter * Update _clients/data-prepper/migrate-open-distro.md Co-authored-by: Heather Halter * Removing OpenDistro file. Signed-off-by: carolxob * Reconcile branches. Signed-off-by: carolxob * Apply suggestions from code review Made changes based on feedback. Co-authored-by: Nate Bower Signed-off-by: carolxob Co-authored-by: Heather Halter Co-authored-by: Nate Bower --- .../configure-logstash-data-prepper.md | 44 +++++++++++++++++++ _clients/data-prepper/migrate-open-distro.md | 2 +- 2 files changed, 45 insertions(+), 1 deletion(-) create mode 100644 _clients/data-prepper/configure-logstash-data-prepper.md diff --git a/_clients/data-prepper/configure-logstash-data-prepper.md b/_clients/data-prepper/configure-logstash-data-prepper.md new file mode 100644 index 0000000000..b58b531b3a --- /dev/null +++ b/_clients/data-prepper/configure-logstash-data-prepper.md @@ -0,0 +1,44 @@ +--- +layout: default +title: Configure Logstash for Data Prepper +parent: Data Prepper +nav_order: 2 +--- +# Configure Logstash for Data Prepper +You can run Data Prepper with a Logstash configuration. + +As mentioned in the [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting-started) guide, you'll need to configure Data Prepper with a pipeline using a `pipelines.yaml` file. + +Alternatively, if you have a Logstash configuration `logstash.conf` to configure Data Prepper instead of `pipelines.yaml`. + +## Supported plugins + +As of the Data Prepper 1.2 release, the following plugins from the Logstash configuration are supported: +* HTTP Input plugin +* Grok Filter plugin +* Elasticsearch Output plugin +* Amazon Elasticsearch Output plugin + +## Limitations +* Apart from the supported plugins, all other plugins from the Logstash configuration will throw an `Exception` and fail to run. +* Conditionals in the Logstash configuration are not supported as of the Data Prepper 1.2 release. + +## Running Data Prepper with a Logstash configuration + +1. To install Data Prepper's Docker image, see the _Installation_ section in the [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting_started) guide. + +2. Run the Docker image installed in Step 1 by supplying your `logstash.conf` configuration. + +``` +docker run --name data-prepper -p 4900:4900 -v ${PWD}/logstash.conf:/usr/share/data-prepper/pipelines.conf opensearchproject/data-prepper:latest pipelines.conf +``` + +The `logstash.conf` file is converted to `logstash.yaml` by mapping the plugins and attributes in the Logstash configuration to the corresponding plugins and attributes in Data Prepper. +You can find the converted `logstash.yaml` file in the same directory where you stored `logstash.conf`. + + +The following output in your terminal indicates that Data Prepper is running correctly: + +``` +INFO org.opensearch.dataprepper.pipeline.ProcessWorker - log-pipeline Worker: No records received from buffer +``` \ No newline at end of file diff --git a/_clients/data-prepper/migrate-open-distro.md b/_clients/data-prepper/migrate-open-distro.md index 3d1c3eb558..e6f1deca02 100644 --- a/_clients/data-prepper/migrate-open-distro.md +++ b/_clients/data-prepper/migrate-open-distro.md @@ -22,4 +22,4 @@ In your Data Prepper Docker configuration, adjust `amazon/opendistro-for-elastic ## Next steps -For more information about Data Prepper configurations, see [Getting Started with Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/get-started/). \ No newline at end of file +For more information about Data Prepper configurations, see [Getting Started with Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/get-started/).