User facing documentation for Logstash on ECK #6585

robbavey · 2023-03-24T19:40:08Z

Work in Progress docs

Includes

Rough quickstart guide
How to Configure Logstash on ECK, including
- Logstash Configuration
- Pipeline Configuration
- Using ElasticsearchRef
- Scaling Logstash
- Exposing Services
Sample recipes for configuring pipelines in different ways
Advanced Configuration - setting JVM options
Limitations of the Technical Preview

kaisecheng

I think we can update the container version in all recipes from 8.6.1 to 8.7.0, especially Logstash stack monitoring only works with 8.7+

For the todo, we will need to cover pipeline reload use case.

kaisecheng · 2023-04-03T20:58:14Z

config/recipes/logstash/logstash-eck.yaml

+          }
+        }
+        output {
+          stdout { codec => rubydebug }


As elasticsearch cluster is included in the sample, I think the pipeline can have es-output example

elasticsearch { hosts => [ "${DEFAULT_ELASTICSEARCH_HOSTS}" ] user => "${DEFAULT_ELASTICSEARCH_USER}" password => "${DEFAULT_ELASTICSEARCH_PASSWORD}" cacert => "${DEFAULT_ELASTICSEARCH_CA_CERTS}" }

kaisecheng · 2023-04-03T20:59:32Z

config/recipes/logstash/logstash-multi.yaml

+            send_to => 'prod'
+          }
+          pipeline {
+            send_to => 'qa'


👍 nice pipeline-to-pipeline example

Nearly! I forgot to remove some old pipelines 🤦

docs/advanced-topics/stack-monitoring.asciidoc

kaisecheng · 2023-04-17T18:04:16Z

need a section for es-output to point users to customize users and roles

mashhurs · 2023-04-26T14:56:39Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

+=== Single Pipeline defined in CRD
+
+[source,sh,subs="attributes"]
+----
+kubectl apply -f {logstash_recipes}/logstash-eck.yaml
+----
+
+Deploys Logstash with a single pipeline defined in the CRD


Is it intentional to repeat this Single Pipeline defined in CRD section?

It is not 🤦

mashhurs · 2023-04-26T15:24:24Z

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

+* <<{p}-logstash-technical-preview-limitations,Technical Preview Limitations>>
+
+
+NOTE: Running Logstash on ECK is compatible only with Logstash 8.7+.


Do we need to remove this? I do see LS 8.6.1 versions are in settings (ex: logstash.yml) and as I understood stack monitoring for Logstash requires 8.7.0+.
Or, we can change all configs version to 8.7.0 to align with this note.

* In the init container: copy the Elasticsearch configuration, and then create links. * Set default security context for Elasticsearch containers, including the sidecars. * Add E2E check for securityContext.

…ic#6724) * Introduce HasWarnings interface * Update samples in config/recipes * Update example in docs * Update CRD/API documentation * Update Helm Charts * Add checkPolicyID for next major release

* Update module go.elastic.co/apm/v2 to v2.4.1 * Update module go.elastic.co/apm/module/apmzap/v2 to v2.4.1 * Update module go.elastic.co/apm/module/apmhttp/v2 to v2.4.1 * Update module go.elastic.co/apm/module/apmelasticsearch/v2 to v2.4.1 --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Thibault Richard <[email protected]>

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

pebrc

Did a quick pass LGTM, found a few nits only.

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

alaudazzi · 2023-04-28T11:26:04Z

I referenced the new file from the Orchestrating Elastic Stack applications landing page.

alaudazzi

I had a second pass and left a few editing suggestions.

docs/advanced-topics/stack-monitoring.asciidoc

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

config/recipes/logstash/README.asciidoc

thbkrkr · 2023-04-28T10:18:35Z

config/recipes/logstash/README.asciidoc

+
+Deploys Logstash with the pipeline defined in a secret and referred to via `pipelinesRef`
+
+===== Pipeline as mounted volume - `logstash-pipelinevolume.yaml`


Proposal which implies to rename the file.

Suggested change

===== Pipeline as mounted volume - `logstash-pipelinevolume.yaml`

===== Pipeline as mounted volume - `logstash-pipeline-as-volume.yaml`

config/recipes/logstash/README.asciidoc

docs/orchestrating-elastic-stack-applications/logstash.asciidoc

config/recipes/logstash/logstash-es-role.yaml

This commit introduces a technical preview of the Logstash Operator for ECK The Logstash operator introduces a Logstash CRD: This operator provides support for: * Defining logstash.yml in config or configRef sections of the CRD * Integration with Elasticsearch clusters via the use of elasticsearchRefs, and environment variable substitution to introduce those elasticsearch references into logstash pipelines. * Definition of pipelines.yml in pipelines or pipelinesRef sections of the CRD with support for pipeline definition in volume mounts * Support for multiple pipeline and pipeline->pipeline configurations * Support for automatic pipeline reload in logstash pods when a pipeline change is detected without triggering a full restart of the pod. * Stack monitoring support via sending metrics and logs to a monitoring elasticsearch cluster via the use of monitoring.logs.elasticsearchRefs and monitoring.metrics.elasticsearchRefs * Support for defining multiple services for logstash plugins. Logstash nodes are created as StatefulSets - we expect in later versions of the logstash operator to support persistence in Logstash nodes, including persistent queues and dead letter queues. A work in progress PR includes documentation and recipes on how to use this logstash operator. There are also samples in this PR located under config/samples/logstash/* Co-authored-by: Michael Morello <[email protected]> Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Kaise Cheng <[email protected]> Co-authored-by: kaisecheng <[email protected]> Co-authored-by: Michael Morello <[email protected]> Co-authored-by: Thibault Richard <[email protected]> Co-authored-by: Peter Brachwitz <[email protected]>

) This commit ensures that we take into account the volumes provided by the user to append the default elasticsearch-data volumeMount.

This adds a new flag `--container-repository` to the operator to be able to specify a global container repository. With this it is now possible to use DockerHub images for example: `--container-registry docker.io --container-repository=elastic`. --------- Co-authored-by: Peter Brachwitz <[email protected]>

This adjusts the number of volumes expected from Beats sidecars in the Logstash Stack Monitoring unit tests. Why? Because we don't test PRs with an automatic merge of the main branch (🐛🐞), we missed that the tests in elastic#6732 had to be updated to take into account the changes made by elastic#6703, which adds a new temp volume to the Beats sidecars.

Still a work in progress - will need elasticsearchRef to work fully

Co-authored-by: Thibault Richard <[email protected]> Co-authored-by: Arianna Laudazzi <[email protected]> Co-authored-by: Peter Brachwitz <[email protected]>

robbavey · 2023-04-28T16:27:47Z

I've answered all the comments on this PR, and created a new one retargeted to the main branch:

#6743

botelastic bot added the triage label Mar 24, 2023

thbkrkr added the >docs Documentation label Mar 24, 2023

botelastic bot removed the triage label Mar 24, 2023

robbavey requested a review from kaisecheng April 3, 2023 13:25

kaisecheng reviewed Apr 3, 2023

View reviewed changes

robbavey marked this pull request as ready for review April 25, 2023 15:30

mashhurs reviewed Apr 26, 2023

View reviewed changes

robbavey mentioned this pull request Apr 26, 2023

Introduce the Logstash operator for ECK #6732

Merged

barkbay added 2 commits April 27, 2023 11:37

Harden Elasticsearch SecurityContext (elastic#6703)

7a57bbd

* In the init container: copy the Elasticsearch configuration, and then create links. * Set default security context for Elasticsearch containers, including the sidecars. * Add E2E check for securityContext.

[Fleet] Deprecate is_default and is_default_fleet_server flags (elast…

326eb00

…ic#6724) * Introduce HasWarnings interface * Update samples in config/recipes * Update example in docs * Update CRD/API documentation * Update Helm Charts * Add checkPolicyID for next major release

thbkrkr added the v2.8.0 label Apr 27, 2023

pebrc reviewed Apr 28, 2023

View reviewed changes

docs/orchestrating-elastic-stack-applications/logstash.asciidoc Show resolved Hide resolved

pebrc reviewed Apr 28, 2023

View reviewed changes

alaudazzi self-requested a review April 28, 2023 11:26

alaudazzi reviewed Apr 28, 2023

View reviewed changes

thbkrkr reviewed Apr 28, 2023

View reviewed changes

robbavey and others added 4 commits April 28, 2023 16:13

Fix default elasticsearch-data volumeMount configuration (elastic#6725

294f302

) This commit ensures that we take into account the volumes provided by the user to append the default elasticsearch-data volumeMount.

robbavey force-pushed the logstash_doc branch 2 times, most recently from 2d57563 to 16694f1 Compare April 28, 2023 16:11

robbavey added 4 commits April 28, 2023 12:18

First pass

0b7a58a

Added more logstash docs

4f1b102

Add Logstash Recipes

b2e00d4

Still a work in progress - will need elasticsearchRef to work fully

Tidy up readme

90683af

robbavey and others added 16 commits April 28, 2023 12:18

Added more doc

9ecb8cd

Improvements

d14faf0

add recipes for ElasticsearchRefs

f9ba1ff

Responded to code review comments

38de4dd

Updates for elasticsearchRef. A little tidying

a29d9f0

Add role information to elasticsearch integration

f07d6ce

Fix env variable names in example snippets

2f45d19

Fix recipes

add2f2c

Add intro to elasticsearchref example

7aa2a6c

Remove duplicate recipe reference.

5872086

Add the new file to the landing page

6ceb4f9

Apply suggestions from code review

e98acfe

Co-authored-by: Thibault Richard <[email protected]> Co-authored-by: Arianna Laudazzi <[email protected]> Co-authored-by: Peter Brachwitz <[email protected]>

Tidy up recipes after code review comments

301a554

More code review suggestions

232d61b

Missed capitalization

c3b9007

Merge remote-tracking branch 'upstream/main' into logstash_doc

353cac7

robbavey force-pushed the logstash_doc branch from 16694f1 to 353cac7 Compare April 28, 2023 16:22

robbavey closed this Apr 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

User facing documentation for Logstash on ECK #6585

User facing documentation for Logstash on ECK #6585

robbavey commented Mar 24, 2023 •

edited

Loading

kaisecheng left a comment

kaisecheng Apr 3, 2023

kaisecheng Apr 3, 2023

robbavey Apr 3, 2023

kaisecheng commented Apr 17, 2023

mashhurs Apr 26, 2023

robbavey Apr 26, 2023

mashhurs Apr 26, 2023

pebrc left a comment

alaudazzi commented Apr 28, 2023

alaudazzi left a comment

thbkrkr Apr 28, 2023

robbavey commented Apr 28, 2023

		* <<{p}-logstash-technical-preview-limitations,Technical Preview Limitations>>


		NOTE: Running Logstash on ECK is compatible only with Logstash 8.7+.


		Deploys Logstash with the pipeline defined in a secret and referred to via `pipelinesRef`

		===== Pipeline as mounted volume - `logstash-pipelinevolume.yaml`

User facing documentation for Logstash on ECK #6585

User facing documentation for Logstash on ECK #6585

Conversation

robbavey commented Mar 24, 2023 • edited Loading

kaisecheng left a comment

Choose a reason for hiding this comment

kaisecheng Apr 3, 2023

Choose a reason for hiding this comment

kaisecheng Apr 3, 2023

Choose a reason for hiding this comment

robbavey Apr 3, 2023

Choose a reason for hiding this comment

kaisecheng commented Apr 17, 2023

mashhurs Apr 26, 2023

Choose a reason for hiding this comment

robbavey Apr 26, 2023

Choose a reason for hiding this comment

mashhurs Apr 26, 2023

Choose a reason for hiding this comment

pebrc left a comment

Choose a reason for hiding this comment

alaudazzi commented Apr 28, 2023

alaudazzi left a comment

Choose a reason for hiding this comment

thbkrkr Apr 28, 2023

Choose a reason for hiding this comment

robbavey commented Apr 28, 2023

robbavey commented Mar 24, 2023 •

edited

Loading