Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logstash migration guide #1487

Merged
merged 10 commits into from
Nov 7, 2022
44 changes: 44 additions & 0 deletions _clients/data-prepper/configure-logstash-data-prepper.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
layout: default
title: Configure Logstash for Data Prepper
parent: Data Prepper
nav_order: 2
---
# Configure Logstash for Data Prepper
You can run Data Prepper with a Logstash configuration.

As mentioned in the [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting-started) guide, you'll need to configure Data Prepper with a pipeline using a `pipelines.yaml` file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting Started guide --> Getting Started section?


Alternatively, if you have a Logstash configuration `logstash.conf` to configure Data Prepper instead of `pipelines.yaml`.
carolxob marked this conversation as resolved.
Show resolved Hide resolved
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## Supported Plugins
carolxob marked this conversation as resolved.
Show resolved Hide resolved

As of the Data Prepper 1.2 release, the following plugins from the Logstash configuration are supported:
* HTTP Input plugin
* Grok Filter plugin
* Elasticsearch Output plugin
* Amazon Elasticsearch Output plugin

## Limitations
* Apart from the supported plugins, all other plugins from the Logstash configuration will throw an `Exception` and fail to execute.
carolxob marked this conversation as resolved.
Show resolved Hide resolved
* Conditionals in the Logstash configuration are not supported as of Data Prepper 1.2 release.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## Running Data Prepper with Logstash Configuration
carolxob marked this conversation as resolved.
Show resolved Hide resolved

1. To install Data Prepper's docker image, visit the _Installation_ section in [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting_started).
carolxob marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be "To install the Data Prepper Docker image" in order to avoid using punctuation with "Data Prepper"?


2. Run the docker image pulled in Step 1 by supplying your `logstash.conf` configuration.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

```
docker run --name data-prepper -p 4900:4900 -v ${PWD}/logstash.conf:/usr/share/data-prepper/pipelines.conf opensearchproject/data-prepper:latest pipelines.conf
```

The `logstash.conf` file gets converted to `logstash.yaml` by mapping the plugins and attributes in the Logstash configuration to the corresponding plugins and attributes in Data Prepper.
carolxob marked this conversation as resolved.
Show resolved Hide resolved
You can find the converted `logstash.yaml` file in the same directory where you placed `logstash.conf`.
carolxob marked this conversation as resolved.
Show resolved Hide resolved
carolxob marked this conversation as resolved.
Show resolved Hide resolved


The following output in your terminal indicates that Data Prepper is running correctly:

```
INFO org.opensearch.dataprepper.pipeline.ProcessWorker - log-pipeline Worker: No records received from buffer
```
24 changes: 24 additions & 0 deletions _clients/data-prepper/migrate-open-distro.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
layout: default
title: Migrating from Open Distro Data Prepper
parent: Data Prepper
nav_order: 2
---

# Migrating from Open Distro Data Prepper
carolxob marked this conversation as resolved.
Show resolved Hide resolved

With Data Prepper 1.1, there was only one distribution which was through Open Distro. With the 2.0 release, you can now migrate from the Open Distro Data Prepper to OpenSearch Data Prepper.

## Change your Pipeline Configuration

The `elasticsearch` sink has changed to `opensearch`. You will need to change your existing pipeline to use the `opensearch` plugin instead of `elasticsearch`.

Please note that while the plugin is titled `opensearch`, it remains compatible with Open Distro and ElasticSearch 7.x.

## Update Docker Image

The Open Distro Data Prepper Docker image was located at `amazon/opendistro-for-elasticsearch-data-prepper`. You will need to change this value to `opensearchproject/data-prepper`.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## More Information
carolxob marked this conversation as resolved.
Show resolved Hide resolved

You can find more information about Data Prepper configurations by going to the [Getting Started](getting_started.md) guide.
2 changes: 1 addition & 1 deletion _clients/data-prepper/pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
layout: default
title: Pipelines
parent: Data Prepper
nav_order: 2
nav_order: 10
---

# Pipelines
Expand Down