Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logstash migration guide #1487

Merged
merged 10 commits into from
Nov 7, 2022
Merged

Logstash migration guide #1487

merged 10 commits into from
Nov 7, 2022

Conversation

carolxob
Copy link
Contributor

@carolxob carolxob commented Oct 7, 2022

Description

Migrated Logstash Migration guide from DataPrepper doc site to OpenSearch doc site.

Issues Resolved

Resolves #1482

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@carolxob carolxob added backport 2.0 PR: Backport label for v2.0.x 4 - Doc review PR: Doc review in progress backport 2.1 PR: Backport label for 2.1 backport 2.2 PR: Backport label for 2.2 backport 2.3 PR: Backport label for 2.3 data-prepper labels Oct 7, 2022
@carolxob carolxob added this to the 2022-Q4 milestone Oct 7, 2022
@carolxob carolxob requested a review from a team as a code owner October 7, 2022 21:54
@carolxob carolxob self-assigned this Oct 7, 2022
Copy link
Contributor

@hdhalter hdhalter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Caroline! Most of my comments are around context. Since this is part of our documentation, we need to avoid using 'guide'.

_clients/data-prepper/logstash-migration-guide.md Outdated Show resolved Hide resolved
_clients/data-prepper/logstash-migration-guide.md Outdated Show resolved Hide resolved
_clients/data-prepper/logstash-migration-guide.md Outdated Show resolved Hide resolved
_clients/data-prepper/logstash-migration-guide.md Outdated Show resolved Hide resolved
_clients/data-prepper/logstash-migration-guide.md Outdated Show resolved Hide resolved
_clients/data-prepper/migrate-open-distro.md Outdated Show resolved Hide resolved
_clients/data-prepper/migrate-open-distro.md Outdated Show resolved Hide resolved
_clients/data-prepper/migrate-open-distro.md Outdated Show resolved Hide resolved
_clients/data-prepper/migrate-open-distro.md Outdated Show resolved Hide resolved
_clients/data-prepper/migrate-open-distro.md Outdated Show resolved Hide resolved
@carolxob carolxob requested a review from hdhalter October 11, 2022 17:11
@carolxob carolxob added 5 - Editorial review PR: Editorial review in progress and removed 4 - Doc review PR: Doc review in progress labels Oct 12, 2022
Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Naarcha-AWS Please see my changes and comments and let me know if you have any questions. Thanks!

_clients/data-prepper/configure-logstash-data-prepper.md Outdated Show resolved Hide resolved
_clients/data-prepper/configure-logstash-data-prepper.md Outdated Show resolved Hide resolved
_clients/data-prepper/configure-logstash-data-prepper.md Outdated Show resolved Hide resolved
_clients/data-prepper/configure-logstash-data-prepper.md Outdated Show resolved Hide resolved

## Running Data Prepper with Logstash Configuration

1. To install Data Prepper's docker image, visit the _Installation_ section in [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting_started).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be "To install the Data Prepper Docker image" in order to avoid using punctuation with "Data Prepper"?

_clients/data-prepper/configure-logstash-data-prepper.md Outdated Show resolved Hide resolved
_clients/data-prepper/configure-logstash-data-prepper.md Outdated Show resolved Hide resolved
_clients/data-prepper/configure-logstash-data-prepper.md Outdated Show resolved Hide resolved
_clients/data-prepper/migrate-open-distro.md Outdated Show resolved Hide resolved
@natebower natebower removed the 5 - Editorial review PR: Editorial review in progress label Oct 14, 2022
@carolxob carolxob added the 5 - Editorial review PR: Editorial review in progress label Nov 3, 2022
Made changes based on feedback.

Co-authored-by: Nate Bower <[email protected]>
Copy link
Contributor

@hdhalter hdhalter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Caroline, just a few tweaks. I'll take another look once these are addressed. Thanks!

# Configure Logstash for Data Prepper
You can run Data Prepper with a Logstash configuration.

As mentioned in the [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting-started) guide, you'll need to configure Data Prepper with a pipeline using a `pipelines.yaml` file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting Started guide --> Getting Started section?

* Apart from the supported plugins, all other plugins from the Logstash configuration will throw an `Exception` and fail to run.
* Conditionals in the Logstash configuration are not supported as of the Data Prepper 1.2 release.

## Running Data Prepper with a Logstash configuration
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have an intro paragraph here? Like To run Data Prepper with a Logstash configuration, do the following.


## Running Data Prepper with a Logstash configuration

1. To install Data Prepper's Docker image, see the _Installation_ section in the [Getting Started]({{site.url}}{{site.baseurl}}/opensearch/clients/data-prepper/getting_started) guide.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a command here, like "Install the Data Prepper Docker image. For instructions, see ....

Remove "guide".

@carolxob carolxob merged commit 3017f51 into main Nov 7, 2022
@carolxob carolxob deleted the logstash-migration-guide branch November 7, 2022 16:25
opensearch-trigger-bot bot pushed a commit that referenced this pull request Nov 7, 2022
* Added cluster permissions to cluster permissions list.

Signed-off-by: carolxob <[email protected]>

* Revert "Added cluster permissions to cluster permissions list."

This reverts commit 48a9fb5.

* Migrated pipelines and migrating from Open Distro.

Signed-off-by: carolxob <[email protected]>

* Migrated Logstash Migration Guide to doc site.

Signed-off-by: carolxob <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Removing OpenDistro file.

Signed-off-by: carolxob <[email protected]>

* Reconcile branches.

Signed-off-by: carolxob <[email protected]>

* Apply suggestions from code review

Made changes based on feedback.

Co-authored-by: Nate Bower <[email protected]>

Signed-off-by: carolxob <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nate Bower <[email protected]>
(cherry picked from commit 3017f51)
opensearch-trigger-bot bot pushed a commit that referenced this pull request Nov 7, 2022
* Added cluster permissions to cluster permissions list.

Signed-off-by: carolxob <[email protected]>

* Revert "Added cluster permissions to cluster permissions list."

This reverts commit 48a9fb5.

* Migrated pipelines and migrating from Open Distro.

Signed-off-by: carolxob <[email protected]>

* Migrated Logstash Migration Guide to doc site.

Signed-off-by: carolxob <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Removing OpenDistro file.

Signed-off-by: carolxob <[email protected]>

* Reconcile branches.

Signed-off-by: carolxob <[email protected]>

* Apply suggestions from code review

Made changes based on feedback.

Co-authored-by: Nate Bower <[email protected]>

Signed-off-by: carolxob <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nate Bower <[email protected]>
(cherry picked from commit 3017f51)
opensearch-trigger-bot bot pushed a commit that referenced this pull request Nov 7, 2022
* Added cluster permissions to cluster permissions list.

Signed-off-by: carolxob <[email protected]>

* Revert "Added cluster permissions to cluster permissions list."

This reverts commit 48a9fb5.

* Migrated pipelines and migrating from Open Distro.

Signed-off-by: carolxob <[email protected]>

* Migrated Logstash Migration Guide to doc site.

Signed-off-by: carolxob <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Removing OpenDistro file.

Signed-off-by: carolxob <[email protected]>

* Reconcile branches.

Signed-off-by: carolxob <[email protected]>

* Apply suggestions from code review

Made changes based on feedback.

Co-authored-by: Nate Bower <[email protected]>

Signed-off-by: carolxob <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nate Bower <[email protected]>
(cherry picked from commit 3017f51)
opensearch-trigger-bot bot pushed a commit that referenced this pull request Nov 7, 2022
* Added cluster permissions to cluster permissions list.

Signed-off-by: carolxob <[email protected]>

* Revert "Added cluster permissions to cluster permissions list."

This reverts commit 48a9fb5.

* Migrated pipelines and migrating from Open Distro.

Signed-off-by: carolxob <[email protected]>

* Migrated Logstash Migration Guide to doc site.

Signed-off-by: carolxob <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Removing OpenDistro file.

Signed-off-by: carolxob <[email protected]>

* Reconcile branches.

Signed-off-by: carolxob <[email protected]>

* Apply suggestions from code review

Made changes based on feedback.

Co-authored-by: Nate Bower <[email protected]>

Signed-off-by: carolxob <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nate Bower <[email protected]>
(cherry picked from commit 3017f51)
carolxob added a commit that referenced this pull request Nov 7, 2022
* Added cluster permissions to cluster permissions list.

Signed-off-by: carolxob <[email protected]>

* Revert "Added cluster permissions to cluster permissions list."

This reverts commit 48a9fb5.

* Migrated pipelines and migrating from Open Distro.

Signed-off-by: carolxob <[email protected]>

* Migrated Logstash Migration Guide to doc site.

Signed-off-by: carolxob <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Removing OpenDistro file.

Signed-off-by: carolxob <[email protected]>

* Reconcile branches.

Signed-off-by: carolxob <[email protected]>

* Apply suggestions from code review

Made changes based on feedback.

Co-authored-by: Nate Bower <[email protected]>

Signed-off-by: carolxob <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nate Bower <[email protected]>
(cherry picked from commit 3017f51)

Co-authored-by: Caroline <[email protected]>
carolxob added a commit that referenced this pull request Nov 7, 2022
* Added cluster permissions to cluster permissions list.

Signed-off-by: carolxob <[email protected]>

* Revert "Added cluster permissions to cluster permissions list."

This reverts commit 48a9fb5.

* Migrated pipelines and migrating from Open Distro.

Signed-off-by: carolxob <[email protected]>

* Migrated Logstash Migration Guide to doc site.

Signed-off-by: carolxob <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Removing OpenDistro file.

Signed-off-by: carolxob <[email protected]>

* Reconcile branches.

Signed-off-by: carolxob <[email protected]>

* Apply suggestions from code review

Made changes based on feedback.

Co-authored-by: Nate Bower <[email protected]>

Signed-off-by: carolxob <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nate Bower <[email protected]>
(cherry picked from commit 3017f51)

Co-authored-by: Caroline <[email protected]>
carolxob added a commit that referenced this pull request Nov 7, 2022
* Added cluster permissions to cluster permissions list.

Signed-off-by: carolxob <[email protected]>

* Revert "Added cluster permissions to cluster permissions list."

This reverts commit 48a9fb5.

* Migrated pipelines and migrating from Open Distro.

Signed-off-by: carolxob <[email protected]>

* Migrated Logstash Migration Guide to doc site.

Signed-off-by: carolxob <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Removing OpenDistro file.

Signed-off-by: carolxob <[email protected]>

* Reconcile branches.

Signed-off-by: carolxob <[email protected]>

* Apply suggestions from code review

Made changes based on feedback.

Co-authored-by: Nate Bower <[email protected]>

Signed-off-by: carolxob <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nate Bower <[email protected]>
(cherry picked from commit 3017f51)

Co-authored-by: Caroline <[email protected]>
carolxob added a commit that referenced this pull request Nov 7, 2022
* Added cluster permissions to cluster permissions list.

Signed-off-by: carolxob <[email protected]>

* Revert "Added cluster permissions to cluster permissions list."

This reverts commit 48a9fb5.

* Migrated pipelines and migrating from Open Distro.

Signed-off-by: carolxob <[email protected]>

* Migrated Logstash Migration Guide to doc site.

Signed-off-by: carolxob <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Update _clients/data-prepper/migrate-open-distro.md

Co-authored-by: Heather Halter <[email protected]>

* Removing OpenDistro file.

Signed-off-by: carolxob <[email protected]>

* Reconcile branches.

Signed-off-by: carolxob <[email protected]>

* Apply suggestions from code review

Made changes based on feedback.

Co-authored-by: Nate Bower <[email protected]>

Signed-off-by: carolxob <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nate Bower <[email protected]>
(cherry picked from commit 3017f51)

Co-authored-by: Caroline <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5 - Editorial review PR: Editorial review in progress backport 2.0 PR: Backport label for v2.0.x backport 2.1 PR: Backport label for 2.1 backport 2.2 PR: Backport label for 2.2 backport 2.3 PR: Backport label for 2.3 data-prepper
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Data Prepper Logstash Migration Guide
5 participants