-
Notifications
You must be signed in to change notification settings - Fork 508
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Logstash migration guide #1487
Merged
Merged
Logstash migration guide #1487
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
e896071
Added cluster permissions to cluster permissions list.
carolxob 12edb4a
Revert "Added cluster permissions to cluster permissions list."
carolxob 0790bbf
Migrated pipelines and migrating from Open Distro.
carolxob 09870b4
Migrated Logstash Migration Guide to doc site.
carolxob 8622d56
Update _clients/data-prepper/migrate-open-distro.md
carolxob 53bcbfa
Update _clients/data-prepper/migrate-open-distro.md
carolxob e210d4c
Removing OpenDistro file.
carolxob 9c81f7f
Reconcile branches.
carolxob cad2334
Apply suggestions from code review
carolxob 94501cb
Merge branch 'main' into logstash-migration-guide
carolxob File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
--- | ||
layout: default | ||
title: Configure Logstash for Data Prepper | ||
parent: Data Prepper | ||
nav_order: 2 | ||
--- | ||
# Configure Logstash for Data Prepper | ||
You can run Data Prepper with a Logstash configuration. | ||
|
||
As mentioned in the [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting-started) guide, you'll need to configure Data Prepper with a pipeline using a `pipelines.yaml` file. | ||
|
||
Alternatively, if you have a Logstash configuration `logstash.conf` to configure Data Prepper instead of `pipelines.yaml`. | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## Supported Plugins | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
As of the Data Prepper 1.2 release, the following plugins from the Logstash configuration are supported: | ||
* HTTP Input plugin | ||
* Grok Filter plugin | ||
* Elasticsearch Output plugin | ||
* Amazon Elasticsearch Output plugin | ||
|
||
## Limitations | ||
* Apart from the supported plugins, all other plugins from the Logstash configuration will throw an `Exception` and fail to execute. | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* Conditionals in the Logstash configuration are not supported as of Data Prepper 1.2 release. | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## Running Data Prepper with Logstash Configuration | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
1. To install Data Prepper's docker image, visit the _Installation_ section in [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting_started). | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can this be "To install the Data Prepper Docker image" in order to avoid using punctuation with "Data Prepper"? |
||
|
||
2. Run the docker image pulled in Step 1 by supplying your `logstash.conf` configuration. | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
``` | ||
docker run --name data-prepper -p 4900:4900 -v ${PWD}/logstash.conf:/usr/share/data-prepper/pipelines.conf opensearchproject/data-prepper:latest pipelines.conf | ||
``` | ||
|
||
The `logstash.conf` file gets converted to `logstash.yaml` by mapping the plugins and attributes in the Logstash configuration to the corresponding plugins and attributes in Data Prepper. | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
You can find the converted `logstash.yaml` file in the same directory where you placed `logstash.conf`. | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
The following output in your terminal indicates that Data Prepper is running correctly: | ||
|
||
``` | ||
INFO org.opensearch.dataprepper.pipeline.ProcessWorker - log-pipeline Worker: No records received from buffer | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
--- | ||
layout: default | ||
title: Migrating from Open Distro Data Prepper | ||
parent: Data Prepper | ||
nav_order: 2 | ||
--- | ||
|
||
# Migrating from Open Distro Data Prepper | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
With Data Prepper 1.1, there was only one distribution which was through Open Distro. With the 2.0 release, you can now migrate from the Open Distro Data Prepper to OpenSearch Data Prepper. | ||
|
||
## Change your Pipeline Configuration | ||
|
||
The `elasticsearch` sink has changed to `opensearch`. You will need to change your existing pipeline to use the `opensearch` plugin instead of `elasticsearch`. | ||
|
||
Please note that while the plugin is titled `opensearch`, it remains compatible with Open Distro and ElasticSearch 7.x. | ||
|
||
## Update Docker Image | ||
|
||
The Open Distro Data Prepper Docker image was located at `amazon/opendistro-for-elasticsearch-data-prepper`. You will need to change this value to `opensearchproject/data-prepper`. | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## More Information | ||
carolxob marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
You can find more information about Data Prepper configurations by going to the [Getting Started](getting_started.md) guide. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,7 +2,7 @@ | |
layout: default | ||
title: Pipelines | ||
parent: Data Prepper | ||
nav_order: 2 | ||
nav_order: 10 | ||
--- | ||
|
||
# Pipelines | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Getting Started guide --> Getting Started section?