Skip to content

Commit

Permalink
Logstash migration guide (opensearch-project#1487)
Browse files Browse the repository at this point in the history
* Added cluster permissions to cluster permissions list.

Signed-off-by: carolxob <[email protected]>

* Revert "Added cluster permissions to cluster permissions list."

This reverts commit 48a9fb5.

* Migrated pipelines and migrating from Open Distro.

Signed-off-by: carolxob <[email protected]>

* Migrated Logstash Migration Guide to doc site.

Signed-off-by: carolxob <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Removing OpenDistro file.

Signed-off-by: carolxob <[email protected]>

* Reconcile branches.

Signed-off-by: carolxob <[email protected]>

* Apply suggestions from code review

Made changes based on feedback.

Co-authored-by: Nate Bower <[email protected]>

Signed-off-by: carolxob <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nate Bower <[email protected]>
  • Loading branch information
3 people authored Nov 7, 2022
1 parent f94d071 commit 3017f51
Show file tree
Hide file tree
Showing 2 changed files with 45 additions and 1 deletion.
44 changes: 44 additions & 0 deletions _clients/data-prepper/configure-logstash-data-prepper.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
layout: default
title: Configure Logstash for Data Prepper
parent: Data Prepper
nav_order: 2
---
# Configure Logstash for Data Prepper
You can run Data Prepper with a Logstash configuration.

As mentioned in the [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting-started) guide, you'll need to configure Data Prepper with a pipeline using a `pipelines.yaml` file.

Alternatively, if you have a Logstash configuration `logstash.conf` to configure Data Prepper instead of `pipelines.yaml`.

## Supported plugins

As of the Data Prepper 1.2 release, the following plugins from the Logstash configuration are supported:
* HTTP Input plugin
* Grok Filter plugin
* Elasticsearch Output plugin
* Amazon Elasticsearch Output plugin

## Limitations
* Apart from the supported plugins, all other plugins from the Logstash configuration will throw an `Exception` and fail to run.
* Conditionals in the Logstash configuration are not supported as of the Data Prepper 1.2 release.

## Running Data Prepper with a Logstash configuration

1. To install Data Prepper's Docker image, see the _Installation_ section in the [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting_started) guide.

2. Run the Docker image installed in Step 1 by supplying your `logstash.conf` configuration.

```
docker run --name data-prepper -p 4900:4900 -v ${PWD}/logstash.conf:/usr/share/data-prepper/pipelines.conf opensearchproject/data-prepper:latest pipelines.conf
```

The `logstash.conf` file is converted to `logstash.yaml` by mapping the plugins and attributes in the Logstash configuration to the corresponding plugins and attributes in Data Prepper.
You can find the converted `logstash.yaml` file in the same directory where you stored `logstash.conf`.


The following output in your terminal indicates that Data Prepper is running correctly:

```
INFO org.opensearch.dataprepper.pipeline.ProcessWorker - log-pipeline Worker: No records received from buffer
```
2 changes: 1 addition & 1 deletion _clients/data-prepper/migrate-open-distro.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ In your Data Prepper Docker configuration, adjust `amazon/opendistro-for-elastic

## Next steps

For more information about Data Prepper configurations, see [Getting Started with Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/get-started/).
For more information about Data Prepper configurations, see [Getting Started with Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/get-started/).

0 comments on commit 3017f51

Please sign in to comment.