Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logstash migration guide #1487

Merged
merged 10 commits into from
Nov 7, 2022
Merged

Logstash migration guide #1487

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
e896071
Added cluster permissions to cluster permissions list.
carolxob Oct 3, 2022
12edb4a
Revert "Added cluster permissions to cluster permissions list."
carolxob Oct 3, 2022
0790bbf
Migrated pipelines and migrating from Open Distro.
carolxob Oct 7, 2022
09870b4
Migrated Logstash Migration Guide to doc site.
carolxob Oct 7, 2022
8622d56
Update _clients/data-prepper/migrate-open-distro.md
carolxob Oct 10, 2022
53bcbfa
Update _clients/data-prepper/migrate-open-distro.md
carolxob Oct 10, 2022
e210d4c
Removing OpenDistro file.
carolxob Oct 11, 2022
9c81f7f
Reconcile branches.
carolxob Oct 11, 2022
cad2334
Apply suggestions from code review
carolxob Nov 3, 2022
94501cb
Merge branch 'main' into logstash-migration-guide
carolxob Nov 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions _clients/data-prepper/configure-logstash-data-prepper.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
layout: default
title: Configure Logstash for Data Prepper
parent: Data Prepper
nav_order: 2
---
# Configure Logstash for Data Prepper
You can run Data Prepper with a Logstash configuration.

As mentioned in the [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting-started) guide, you'll need to configure Data Prepper with a pipeline using a `pipelines.yaml` file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting Started guide --> Getting Started section?


Alternatively, if you have a Logstash configuration `logstash.conf` to configure Data Prepper instead of `pipelines.yaml`.
carolxob marked this conversation as resolved.
Show resolved Hide resolved
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## Supported plugins

As of the Data Prepper 1.2 release, the following plugins from the Logstash configuration are supported:
* HTTP Input plugin
* Grok Filter plugin
* Elasticsearch Output plugin
* Amazon Elasticsearch Output plugin

## Limitations
* Apart from the supported plugins, all other plugins from the Logstash configuration will throw an `Exception` and fail to run.
* Conditionals in the Logstash configuration are not supported as of the Data Prepper 1.2 release.

## Running Data Prepper with a Logstash configuration
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have an intro paragraph here? Like To run Data Prepper with a Logstash configuration, do the following.


1. To install Data Prepper's Docker image, see the _Installation_ section in the [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting_started) guide.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a command here, like "Install the Data Prepper Docker image. For instructions, see ....

Remove "guide".


2. Run the Docker image installed in Step 1 by supplying your `logstash.conf` configuration.

```
docker run --name data-prepper -p 4900:4900 -v ${PWD}/logstash.conf:/usr/share/data-prepper/pipelines.conf opensearchproject/data-prepper:latest pipelines.conf
```

The `logstash.conf` file is converted to `logstash.yaml` by mapping the plugins and attributes in the Logstash configuration to the corresponding plugins and attributes in Data Prepper.
You can find the converted `logstash.yaml` file in the same directory where you stored `logstash.conf`.


The following output in your terminal indicates that Data Prepper is running correctly:

```
INFO org.opensearch.dataprepper.pipeline.ProcessWorker - log-pipeline Worker: No records received from buffer
```
2 changes: 1 addition & 1 deletion _clients/data-prepper/migrate-open-distro.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ In your Data Prepper Docker configuration, adjust `amazon/opendistro-for-elastic

## Next steps

For more information about Data Prepper configurations, see [Getting Started with Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/get-started/).
For more information about Data Prepper configurations, see [Getting Started with Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/get-started/).