Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Prepper ToC Update #2514

Merged
merged 45 commits into from
Feb 3, 2023
Merged

Data Prepper ToC Update #2514

merged 45 commits into from
Feb 3, 2023

Conversation

carolxob
Copy link
Contributor

@carolxob carolxob commented Jan 27, 2023

Signed-off-by: carolxob [email protected]

Description

Modifies Data Prepper documentation table of contents for accuracy and usability.

Issues Resolved

Closes issue #2453.

Modifies existing ToC to look like the following:

Getting started
Using for common scenarios
  - Trace analytics [pending review]
  - Log analytics
  - S3 log analytics [removed until content exists]
  - Metrics ingestion [removed until content exists]
Pipelines
  - Pipeline options [Placeholder, content moved to Authoring pipelines page]
  - Expression syntax
  - Sources
  - Buffers
  - Processors
  - Sinks
Administrating Data Prepper
  - Configuring Data Data Prepper
  - Peer forwarder [pending review]
  - Core APIs
  - Log4j configuration
  - Monitoring Data Prepper
Migrating from Logstash
Migrating from Open Distro Data Prepper

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@carolxob carolxob added 2 - In progress Issue/PR: The issue or PR is in progress. data-prepper labels Jan 27, 2023
@carolxob carolxob requested a review from a team as a code owner January 27, 2023 17:05
@carolxob carolxob self-assigned this Jan 27, 2023
@carolxob carolxob changed the title Creating PR with first file. Data Prepper ToC Update Jan 27, 2023
Signed-off-by: carolxob <[email protected]>
Signed-off-by: carolxob <[email protected]>
Signed-off-by: carolxob <[email protected]>
Signed-off-by: carolxob <[email protected]>
@carolxob
Copy link
Contributor Author

@dlvenable Where should the Pipelines section fit in this structure? Right now, it's showing as last in the TOC, but can be moved to wherever it should fit.

@carolxob carolxob added 4 - Doc review PR: Doc review in progress and removed 2 - In progress Issue/PR: The issue or PR is in progress. labels Jan 31, 2023
Copy link
Collaborator

@kolchfa-aws kolchfa-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Left some suggestions.

_data-prepper/getting-started.md Outdated Show resolved Hide resolved
_data-prepper/migrating-from-logstash-data-prepper.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/sources/s3.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/sources/s3.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/pipelines.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/pipelines.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/pipelines.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/pipelines.md Outdated Show resolved Hide resolved
Copy link
Contributor

@vagimeli vagimeli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minimal comment. Good to go to editorial once doc review feedback is addressed.

_data-prepper/getting-started.md Outdated Show resolved Hide resolved
_data-prepper/migrating-from-logstash-data-prepper.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/sources/s3.md Outdated Show resolved Hide resolved
@carolxob carolxob merged commit 0249991 into main Feb 3, 2023
Naarcha-AWS added a commit that referenced this pull request Feb 3, 2023
* Creating PR with first file.

Signed-off-by: carolxob <[email protected]>

* Adding newly created files to PR.

Signed-off-by: carolxob <[email protected]>

* Reorganized files and added appropriate metadata to map ToC correctly.

Signed-off-by: carolxob <[email protected]>

* Moved Authoring pipelines page.

Signed-off-by: carolxob <[email protected]>

* Minor ToC updates.

Signed-off-by: carolxob <[email protected]>

* Minor ToC updates to Sources section for Data Prepper.

Signed-off-by: carolxob <[email protected]>

* Updated Buffers section under Data Prepper.

Signed-off-by: carolxob <[email protected]>

* Minor update to otelmetricssource.

Signed-off-by: carolxob <[email protected]>

* Restructured ToC in Processors section for Data Prepper.

Signed-off-by: carolxob <[email protected]>

* Minor filename change.

Signed-off-by: carolxob <[email protected]>

* Adjustments to metadata in ToC.

Signed-off-by: carolxob <[email protected]>

* Minor edit.

Signed-off-by: carolxob <[email protected]>

* Fixed nav order in metadata.

Signed-off-by: carolxob <[email protected]>

* Minor edit.

Signed-off-by: carolxob <[email protected]>

* Minor update top metadata for ToC.

Signed-off-by: carolxob <[email protected]>

* Adjustmenets to Toc order.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to Sinks section.

Signed-off-by: carolxob <[email protected]>

* Adjustements to high level ToC.

Signed-off-by: carolxob <[email protected]>

* Minor adjustement to Pipelines.md

Signed-off-by: carolxob <[email protected]>

* Minor update.

Signed-off-by: carolxob <[email protected]>

* Slight reorganization. Removed two placeholder pages for now.

Signed-off-by: carolxob <[email protected]>

* Removed a page and replaced with pipelines content.

Signed-off-by: carolxob <[email protected]>

* Minor changes/additions to content for placeholder pages.

Signed-off-by: carolxob <[email protected]>

* Minor update to page link.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <[email protected]>

* Minor edits.

Signed-off-by: carolxob <[email protected]>

* Removed /clients from redirects to correct nav order.

Signed-off-by: carolxob <[email protected]>

* Minor edits.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments.

Signed-off-by: carolxob <[email protected]>

* Minor adjustment ot metadata.

Signed-off-by: carolxob <[email protected]>

* TOC link fixes

Signed-off-by: Naarcha-AWS <[email protected]>

* Changed page name.

Signed-off-by: carolxob <[email protected]>

* Corrected references to Peer Forwarder.

Signed-off-by: carolxob <[email protected]>

* Renamed Data Prepper folder.

Signed-off-by: carolxob <[email protected]>

* Minor updates to phrasing and capitalization.

Signed-off-by: carolxob <[email protected]>

* Minor phrasing update.

Signed-off-by: carolxob <[email protected]>

* Minor phrasing update.

Signed-off-by: carolxob <[email protected]>

* Minor change.

Signed-off-by: carolxob <[email protected]>

* Minor change to change S3 Source to S3Source.

Signed-off-by: carolxob <[email protected]>

* Updated references to peer forwarder and changed capitalization.

Signed-off-by: carolxob <[email protected]>

* Updated capitalization for peer forwarder.

Signed-off-by: carolxob <[email protected]>

* Made edits based on doc review feedback.

Signed-off-by: carolxob <[email protected]>

* Update to one word.

Signed-off-by: carolxob <[email protected]>

---------

Signed-off-by: carolxob <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
@carolxob carolxob added backport 2.0 PR: Backport label for v2.0.x backport 2.1 PR: Backport label for 2.1 backport 2.2 PR: Backport label for 2.2 backport 2.3 PR: Backport label for 2.3 backport 2.4 PR: Backport label for 2.4 backport 2.5 PR: Backport label for 2.5 labels Feb 6, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.0 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.0 2.0
# Navigate to the new working tree
cd .worktrees/backport-2.0
# Create a new branch
git switch --create backport/backport-2514-to-2.0
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0249991f7693be810a8d152eaa1054c115a87dea
# Push it to GitHub
git push --set-upstream origin backport/backport-2514-to-2.0
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.0

Then, create a pull request where the base branch is 2.0 and the compare/head branch is backport/backport-2514-to-2.0.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.5 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.5 2.5
# Navigate to the new working tree
cd .worktrees/backport-2.5
# Create a new branch
git switch --create backport/backport-2514-to-2.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0249991f7693be810a8d152eaa1054c115a87dea
# Push it to GitHub
git push --set-upstream origin backport/backport-2514-to-2.5
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.5

Then, create a pull request where the base branch is 2.5 and the compare/head branch is backport/backport-2514-to-2.5.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.3 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.3 2.3
# Navigate to the new working tree
cd .worktrees/backport-2.3
# Create a new branch
git switch --create backport/backport-2514-to-2.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0249991f7693be810a8d152eaa1054c115a87dea
# Push it to GitHub
git push --set-upstream origin backport/backport-2514-to-2.3
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.3

Then, create a pull request where the base branch is 2.3 and the compare/head branch is backport/backport-2514-to-2.3.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.4 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.4 2.4
# Navigate to the new working tree
cd .worktrees/backport-2.4
# Create a new branch
git switch --create backport/backport-2514-to-2.4
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0249991f7693be810a8d152eaa1054c115a87dea
# Push it to GitHub
git push --set-upstream origin backport/backport-2514-to-2.4
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.4

Then, create a pull request where the base branch is 2.4 and the compare/head branch is backport/backport-2514-to-2.4.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.2 2.2
# Navigate to the new working tree
cd .worktrees/backport-2.2
# Create a new branch
git switch --create backport/backport-2514-to-2.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0249991f7693be810a8d152eaa1054c115a87dea
# Push it to GitHub
git push --set-upstream origin backport/backport-2514-to-2.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.2

Then, create a pull request where the base branch is 2.2 and the compare/head branch is backport/backport-2514-to-2.2.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.1 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.1 2.1
# Navigate to the new working tree
cd .worktrees/backport-2.1
# Create a new branch
git switch --create backport/backport-2514-to-2.1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0249991f7693be810a8d152eaa1054c115a87dea
# Push it to GitHub
git push --set-upstream origin backport/backport-2514-to-2.1
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.1

Then, create a pull request where the base branch is 2.1 and the compare/head branch is backport/backport-2514-to-2.1.

carolxob added a commit that referenced this pull request Feb 6, 2023
* Creating PR with first file.

Signed-off-by: carolxob <[email protected]>

* Adding newly created files to PR.

Signed-off-by: carolxob <[email protected]>

* Reorganized files and added appropriate metadata to map ToC correctly.

Signed-off-by: carolxob <[email protected]>

* Moved Authoring pipelines page.

Signed-off-by: carolxob <[email protected]>

* Minor ToC updates.

Signed-off-by: carolxob <[email protected]>

* Minor ToC updates to Sources section for Data Prepper.

Signed-off-by: carolxob <[email protected]>

* Updated Buffers section under Data Prepper.

Signed-off-by: carolxob <[email protected]>

* Minor update to otelmetricssource.

Signed-off-by: carolxob <[email protected]>

* Restructured ToC in Processors section for Data Prepper.

Signed-off-by: carolxob <[email protected]>

* Minor filename change.

Signed-off-by: carolxob <[email protected]>

* Adjustments to metadata in ToC.

Signed-off-by: carolxob <[email protected]>

* Minor edit.

Signed-off-by: carolxob <[email protected]>

* Fixed nav order in metadata.

Signed-off-by: carolxob <[email protected]>

* Minor edit.

Signed-off-by: carolxob <[email protected]>

* Minor update top metadata for ToC.

Signed-off-by: carolxob <[email protected]>

* Adjustmenets to Toc order.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to Sinks section.

Signed-off-by: carolxob <[email protected]>

* Adjustements to high level ToC.

Signed-off-by: carolxob <[email protected]>

* Minor adjustement to Pipelines.md

Signed-off-by: carolxob <[email protected]>

* Minor update.

Signed-off-by: carolxob <[email protected]>

* Slight reorganization. Removed two placeholder pages for now.

Signed-off-by: carolxob <[email protected]>

* Removed a page and replaced with pipelines content.

Signed-off-by: carolxob <[email protected]>

* Minor changes/additions to content for placeholder pages.

Signed-off-by: carolxob <[email protected]>

* Minor update to page link.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <[email protected]>

* Minor edits.

Signed-off-by: carolxob <[email protected]>

* Removed /clients from redirects to correct nav order.

Signed-off-by: carolxob <[email protected]>

* Minor edits.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments.

Signed-off-by: carolxob <[email protected]>

* Minor adjustment ot metadata.

Signed-off-by: carolxob <[email protected]>

* TOC link fixes

Signed-off-by: Naarcha-AWS <[email protected]>

* Changed page name.

Signed-off-by: carolxob <[email protected]>

* Corrected references to Peer Forwarder.

Signed-off-by: carolxob <[email protected]>

* Renamed Data Prepper folder.

Signed-off-by: carolxob <[email protected]>

* Minor updates to phrasing and capitalization.

Signed-off-by: carolxob <[email protected]>

* Minor phrasing update.

Signed-off-by: carolxob <[email protected]>

* Minor phrasing update.

Signed-off-by: carolxob <[email protected]>

* Minor change.

Signed-off-by: carolxob <[email protected]>

* Minor change to change S3 Source to S3Source.

Signed-off-by: carolxob <[email protected]>

* Updated references to peer forwarder and changed capitalization.

Signed-off-by: carolxob <[email protected]>

* Updated capitalization for peer forwarder.

Signed-off-by: carolxob <[email protected]>

* Made edits based on doc review feedback.

Signed-off-by: carolxob <[email protected]>

* Update to one word.

Signed-off-by: carolxob <[email protected]>

---------

Signed-off-by: carolxob <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
(cherry picked from commit 0249991)
carolxob added a commit that referenced this pull request Feb 6, 2023
* Creating PR with first file.

Signed-off-by: carolxob <[email protected]>

* Adding newly created files to PR.

Signed-off-by: carolxob <[email protected]>

* Reorganized files and added appropriate metadata to map ToC correctly.

Signed-off-by: carolxob <[email protected]>

* Moved Authoring pipelines page.

Signed-off-by: carolxob <[email protected]>

* Minor ToC updates.

Signed-off-by: carolxob <[email protected]>

* Minor ToC updates to Sources section for Data Prepper.

Signed-off-by: carolxob <[email protected]>

* Updated Buffers section under Data Prepper.

Signed-off-by: carolxob <[email protected]>

* Minor update to otelmetricssource.

Signed-off-by: carolxob <[email protected]>

* Restructured ToC in Processors section for Data Prepper.

Signed-off-by: carolxob <[email protected]>

* Minor filename change.

Signed-off-by: carolxob <[email protected]>

* Adjustments to metadata in ToC.

Signed-off-by: carolxob <[email protected]>

* Minor edit.

Signed-off-by: carolxob <[email protected]>

* Fixed nav order in metadata.

Signed-off-by: carolxob <[email protected]>

* Minor edit.

Signed-off-by: carolxob <[email protected]>

* Minor update top metadata for ToC.

Signed-off-by: carolxob <[email protected]>

* Adjustmenets to Toc order.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to Sinks section.

Signed-off-by: carolxob <[email protected]>

* Adjustements to high level ToC.

Signed-off-by: carolxob <[email protected]>

* Minor adjustement to Pipelines.md

Signed-off-by: carolxob <[email protected]>

* Minor update.

Signed-off-by: carolxob <[email protected]>

* Slight reorganization. Removed two placeholder pages for now.

Signed-off-by: carolxob <[email protected]>

* Removed a page and replaced with pipelines content.

Signed-off-by: carolxob <[email protected]>

* Minor changes/additions to content for placeholder pages.

Signed-off-by: carolxob <[email protected]>

* Minor update to page link.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <[email protected]>

* Minor edits.

Signed-off-by: carolxob <[email protected]>

* Removed /clients from redirects to correct nav order.

Signed-off-by: carolxob <[email protected]>

* Minor edits.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments to ToC metadata.

Signed-off-by: carolxob <[email protected]>

* Minor adjustments.

Signed-off-by: carolxob <[email protected]>

* Minor adjustment ot metadata.

Signed-off-by: carolxob <[email protected]>

* TOC link fixes

Signed-off-by: Naarcha-AWS <[email protected]>

* Changed page name.

Signed-off-by: carolxob <[email protected]>

* Corrected references to Peer Forwarder.

Signed-off-by: carolxob <[email protected]>

* Renamed Data Prepper folder.

Signed-off-by: carolxob <[email protected]>

* Minor updates to phrasing and capitalization.

Signed-off-by: carolxob <[email protected]>

* Minor phrasing update.

Signed-off-by: carolxob <[email protected]>

* Minor phrasing update.

Signed-off-by: carolxob <[email protected]>

* Minor change.

Signed-off-by: carolxob <[email protected]>

* Minor change to change S3 Source to S3Source.

Signed-off-by: carolxob <[email protected]>

* Updated references to peer forwarder and changed capitalization.

Signed-off-by: carolxob <[email protected]>

* Updated capitalization for peer forwarder.

Signed-off-by: carolxob <[email protected]>

* Made edits based on doc review feedback.

Signed-off-by: carolxob <[email protected]>

* Update to one word.

Signed-off-by: carolxob <[email protected]>

---------

Signed-off-by: carolxob <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
(cherry picked from commit 0249991)
@hdhalter hdhalter deleted the data-prepper-toc branch March 28, 2024 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
4 - Doc review PR: Doc review in progress backport 2.0 PR: Backport label for v2.0.x backport 2.1 PR: Backport label for 2.1 backport 2.2 PR: Backport label for 2.2 backport 2.3 PR: Backport label for 2.3 backport 2.4 PR: Backport label for 2.4 backport 2.5 PR: Backport label for 2.5 data-prepper
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants