Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logstash migration guide #1487

Merged
merged 10 commits into from
Nov 7, 2022
46 changes: 46 additions & 0 deletions _clients/data-prepper/logstash-migration-guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
layout: default
title: Logstash Migration Guide
parent: Data Prepper
nav_order: 2
---
# Logstash Migration Guide
carolxob marked this conversation as resolved.
Show resolved Hide resolved

This guide describes how to run Data Prepper with a Logstash configuration.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

As mentioned in the [Getting Started](getting_started.md) guide, you need to configure Data Prepper with a pipeline using a `pipelines.yaml` file.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

If you have a Logstash configuration `logstash.conf` file you can use that instead of `pipelines.yaml` to configure Data Prepper.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## Supported Plugins

As of the Data Prepper 1.2 release, the following plugins from the Logstash configuration are supported:
* HTTP Input plugin
* Grok Filter plugin
* Elasticsearch Output plugin
* Amazon Elasticsearch Output plugin

## Limitations
* Apart from the Supported plugins, all other plugins from the Logstash configuration will throw an `Exception` and fail to execute.
carolxob marked this conversation as resolved.
Show resolved Hide resolved
* Conditionals in the Logstash configuration are not supported as of Data Prepper 1.2 release.

## Running Data Prepper with Logstash Configuration

1. To install Data Prepper's docker image, visit the _Installation_ section in the [Getting Started](getting_started.md) guide.
carolxob marked this conversation as resolved.
Show resolved Hide resolved
carolxob marked this conversation as resolved.
Show resolved Hide resolved

2. Run the Docker image pulled in Step 1 by supplying your `logstash.conf`
carolxob marked this conversation as resolved.
Show resolved Hide resolved

```
docker run --name data-prepper -p 4900:4900 -v ${PWD}/logstash.conf:/usr/share/data-prepper/pipelines.conf opensearchproject/data-prepper:latest pipelines.conf
```


The `logstash.conf` gets converted to `logstash.yaml` by mapping the Plugins and attributes in the Logstash configuration to the corresponding ones in Data Prepper.
carolxob marked this conversation as resolved.
Show resolved Hide resolved
You can find the converted `logstash.yaml` in the same directory where you placed `logstash.conf`.
carolxob marked this conversation as resolved.
Show resolved Hide resolved


The following output in your terminal indicates that Data Prepper is running correctly:

```
INFO org.opensearch.dataprepper.pipeline.ProcessWorker - log-pipeline Worker: No records received from buffer
```
24 changes: 24 additions & 0 deletions _clients/data-prepper/migrate-open-distro.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
layout: default
title: Migrating from Open Distro Data Prepper
parent: Data Prepper
nav_order: 2
---

# Migrating from Open Distro Data Prepper
carolxob marked this conversation as resolved.
Show resolved Hide resolved

Beginning with Data Prepper 1.1, there is only one distribution of Data Prepper - Open Search Data Prepper. This document helps existing users migrate from the old Open Distro Data Prepper to OpenSearch Data Prepper.
carolxob marked this conversation as resolved.
Show resolved Hide resolved
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## Change your Pipeline Configuration

The `elasticsearch` sink has changed to `opensearch`. You will need to change your existing pipeline to use the `opensearch` plugin instead of `elasticsearch`.

Please note that, while the plugin is titled `opensearch`, it remains compatible with Open Distro and ElasticSearch 7.x.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## Update Docker Image

The Open Distro Data Prepper Docker image was located at `amazon/opendistro-for-elasticsearch-data-prepper`. You will need to change this value to `opensearchproject/data-prepper`.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## More Information
carolxob marked this conversation as resolved.
Show resolved Hide resolved

You can find more information about Data Prepper configurations by going to the [Getting Started](getting_started.md) guide.
2 changes: 1 addition & 1 deletion _clients/data-prepper/pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
layout: default
title: Pipelines
parent: Data Prepper
nav_order: 2
nav_order: 10
---

# Pipelines
Expand Down