Introduce the Logstash operator for ECK #6732

robbavey · 2023-04-26T22:30:52Z

This commit introduces a technical preview of the Logstash Operator for ECK

The Logstash operator introduces a Logstash CRD:

Look and feel:

Logstash with a simple pipeline, pointing to an elasticsearch instance:

apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-sample
spec:
  count: 1
  version: 8.7.0
  elasticsearchRefs:
    - clusterName: production
      name: elasticsearch-sample
  pipelines:
    - pipeline.id: main
      config.string: |
        input { exec { command => 'uptime' interval => 10 } } 
        output { 
          elasticsearch {
            hosts => [ "${PRODUCTION_ES_HOSTS}" ]
            ssl => true
            cacert => "${PRODUCTION_ES_SSL_CERTIFICATE_AUTHORITY}"
            user => "${PRODUCTION_ES_USER}"
            password => "${PRODUCTION_ES_PASSWORD}"
          } 
        }

Produces the following resources:

 ✗ kubectl get logstash logstash-sample
NAME              AVAILABLE   EXPECTED   AGE     VERSION
logstash-sample   1           1          3m34s   8.7.0

 kubectl tree logstash logstash-sample
NAMESPACE  NAME                                                             READY  REASON  AGE  
pr         Logstash/logstash-sample                                         -              3m57s
pr         ├─Secret/logstash-sample-logstash-es-pr-elasticsearch-sample-ca  -              3m56s
pr         ├─Secret/logstash-sample-ls-config                               -              3m57s
pr         ├─Secret/logstash-sample-ls-pipeline                             -              3m57s
pr         ├─Secret/logstash-sample-pr-elasticsearch-sample-logstash-user   -              3m57s
pr         ├─Service/logstash-sample-ls-api                                 -              3m57s
pr         │ └─EndpointSlice/logstash-sample-ls-api-ndzf5                   -              3m57s
pr         └─StatefulSet/logstash-sample-ls                                 -              3m57s
pr           ├─ControllerRevision/logstash-sample-ls-7d8687f9b              -              3m57s
pr           ├─ControllerRevision/logstash-sample-ls-c6cd667c4              -              3m56s
pr           └─Pod/logstash-sample-ls-0                                     True           3m52s

This operator provides support for:

Defining logstash.yml in config or configRef sections of the CRD
Integration with Elasticsearch clusters via the use of elasticsearchRefs, and environment variable substitution to introduce those elasticsearch references into logstash pipelines.
Definition of pipelines.yml in pipelines or pipelinesRef sections of the CRD with support for pipeline definition in volume mounts
Support for multiple pipeline and pipeline->pipeline configurations
Support for automatic pipeline reload in logstash pods when a pipeline change is detected without triggering a full restart of the pod.
Stack monitoring support via sending metrics and logs to a monitoring elasticsearch cluster via the use of monitoring.logs.elasticsearchRefs and monitoring.metrics.elasticsearchRefs
Support for defining multiple services for logstash plugins.

Logstash nodes are created as StatefulSets - we expect in later versions of the logstash operator to support persistence in Logstash nodes, including persistent queues and dead letter queues.

A work in progress PR includes documentation and recipes on how to use this logstash operator. There are also samples in this PR located under config/samples/logstash/*

Relates: #1453

* Initial Commit of ECK for Logstash Initial commit of a simple operator. The first operator will create: * A Service exposing the logstash metrics API, so it can be used for monitoring purposes, with the option to expose further services if necessary * Secrets holding logstash.yml * A StatefulSet deploying the logstash pods * Pods with default readiness probe The sample logstash yml file as located in config/samples/logstash/logstash_svc.yaml will create: ``` ✗ kubectl tree logstash logstash-sample NAMESPACE NAME READY REASON AGE default Logstash/logstash-sample - 7m3s default ├─Secret/logstash-sample-ls-config - 7m3s default ├─Service/logstash-sample-ls-api - 7m3s default │ └─EndpointSlice/logstash-sample-ls-api-4dl2x - 7m3s default ├─Service/logstash-sample-ls-beats - 7m3s default │ └─EndpointSlice/logstash-sample-ls-beats-rw8sp - 7m3s default ├─Service/logstash-sample-ls-default - 7m3s default │ └─EndpointSlice/logstash-sample-ls-default-6vmhp - 7m3s default └─StatefulSet/logstash-sample-ls - 7m3s default ├─ControllerRevision/logstash-sample-ls-86448964 - 7m3s default ├─Pod/logstash-sample-ls-0 True 7m3s default ├─Pod/logstash-sample-ls-1 True 7m3s default └─Pod/logstash-sample-ls-2 True 7m3s``` And shows status: ``` ✗ kubectl get logstash logstash-sample NAME AVAILABLE EXPECTED AGE VERSION logstash-sample 2 2 2m35s 8.6.1 ``` Co-authored-by: Kaise Cheng <[email protected]> Co-authored-by: kaisecheng <[email protected]> Co-authored-by: Peter Brachwitz <[email protected]> Co-authored-by: Michael Morello <[email protected]>

Following the prior art, this commit integrates stack monitoring for Logstash by adding metricsbeat and filebeat as sidecars to deliver metrics and log to monitoring es cluster There is a sample resource in config/samples/logstash/logstash_stackmonitor.yaml to deploy and test. Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Michael Morello <[email protected]>

This PR allows configuration of Logstash pipeline.yml in CRD. pipeline.yml is an array of map of string/interface{} defining multiple pipelines. `.` in between the key is treated as string. Pipelines can be set from either inline in Pipelines or secret referencing in PipelinesRef. e2e test: TestPipelineConfigRefLogstash, TestPipelineConfigLogstash Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Michael Morello <[email protected]>

This commit adds extra unit tests for the Logstash Controller, including tests for config, controller, pods and services

…pod restart (#6674) This commit adds the ability to reload logstash pipelines when the pipeline changes, without triggering a full restart of the pod, leveraging Logstash's ability to watch pipeline definitions, and reload automatically if a change is discovered. A logstash config directory typically includes logstash.yml, pipelines.yml, jvm.options and log4j2.properties required to run logstash - while logstash can store the contents of a pipeline definition in any location, the pipelines.yml definition file, which states where these definition files are must be in the same config directory as the other setup files. To enables us to have a mixture of copied and generated files in this config directory, an initContainer is used, with a small script to prepare the config directory. The script copies the contents of the /usr/share/logstash/config into the a shared config volume Additionally softlinks are created into this volume from the pipelines.yml and logstash.yml secrets created by the logstash operator. Additionally, we now do not include changes to pipelines in the configuration hash that triggers a pod reload, but instead write the config.reload.automatic: true setting in to logstash.yml Co-authored-by: Peter Brachwitz <[email protected]> Co-authored-by: Thibault Richard <[email protected]>

This commit adds information about Logstash to the set of ECK telemetry Co-authored-by: Thibault Richard <[email protected]>

Fix Logstash service e2e tests This test was failing, because the readiness probe was hitting the port defined by the service rather than the default port. This commit updates the e2e test to set the config to serve the logstash api from the same port that the service specifies. Co-authored-by: Michael Morello <[email protected]>

This commit allows configuration of `elasticsearchRefs` in CRD to associate Elasticsearch clusters to Logstash. The connectivity info/ credentials are passed via environment variable to Logstash container for referencing in Logstash plugins: logstash-output-elasticsearch, logstash-input-elasticsearch and logstash-filter-elasticsearch. Environment variables taking `$clusterName_ES_` as a prefix are available. Co-authored-by: Michael Morello <[email protected]> Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Peter Brachwitz <[email protected]>

thbkrkr · 2023-04-27T09:45:41Z

buildkite test this -f p=gke -m t=TestSamples,t=TestVersionUpgradeOrdering,E2E_TAGS=logstash

🐸 TestLogstashEsOutput 2m27.95s
🐸 TestLogstashPipelineReload 1m2.2s
🐸 TestLogstashServerVersionUpgradeToLatest8x 53.06s
🐸 TestLogstashStackMonitoring 3m47.31s
🐸 TestLogstashWithCustomService 52.84s
🐸 TestLogstashWithCustomServiceAndAmendedApi 52.97s
🐸 TestLogstashWithReworkedApiService 52.83s
🐸 TestMultipleLogstashes 1m1.94s
🐸 TestPipelineConfigLogstash 47s
🐸 TestPipelineConfigRefLogstash 1m17.36s
🐸 TestSingleLogstash 52.84s

🐞 TestSamples

❔ TestVersionUpgradeOrdering (skipped because Logstash isn't supported in 7x)

* Fix the sample tests for logstash service and stack monitoring The stack monitoring tests were failing as the builder was not creating the monitoring elasticsearch references. This was added, as well as a minor refactoring to split the builder into adding metrics and logging monitoring separately. This commit also calculates the the monitoring assocation counts correctly, as previously the metrics association was double counted. This commit also fixes the logstash service sample end to end test - the pipeline id in the yaml definition did not match that in the test. Co-authored-by: Michael Morello <[email protected]>

robbavey · 2023-04-27T13:12:55Z

buildkite test this -f p=gke -m t=TestSamples,t=TestVersionUpgradeOrdering,E2E_TAGS=logstash

🐸 TestSamples 24m0.51s

robbavey · 2023-04-27T14:28:28Z

buildkite test this -f p=gke -m t=TestSamples,t=TestVersionUpgradeOrdering,E2E_TAGS=logstash

robbavey · 2023-04-27T15:24:40Z

@pebrc @barkbay @thbkrkr Fixed the issue with the tests with a commit this morning, and this is now ready for review.

Thanks!

barkbay

Since all the commits in this PR have already been reviewed, and the new E2E tests seem to pass, I think we can merge it as is.

@robbavey please let me know if you think that's not the case and if anything else should be reviewed by the ECK team.

pebrc

LGTM as @barkbay said, this looks like the code we already reviewed. Let us know if there is something specific to look for. Otherwise I am really looking forward to merging this to main. 🎉

pebrc · 2023-04-28T07:55:44Z

config/samples/logstash/logstash_es.yaml

+  elasticsearchRefs:
+    - clusterName: production
+      name: elasticsearch-sample
+#      secretName: external-cloud-es-ref


Remove? I think this is not yet supported.

pkg/apis/logstash/v1alpha1/logstash_types.go

pebrc · 2023-04-28T07:59:03Z

pkg/apis/logstash/v1alpha1/logstash_types.go

+	commonv1.ObjectSelector `json:",omitempty,inline"`
+	// +kubebuilder:validation:Required
+	// +kubebuilder:validation:MinLength=1
+	ClusterName string `json:"clusterName,omitempty"`


Should we add a comment explaining ClusterName?

pebrc · 2023-04-28T08:18:59Z

test/e2e/test/logstash/http_client.go

+// TODO refactor identical to Kibana client
+func NewLogstashClient(logstash v1alpha1.Logstash, k *test.K8sClient) (*http.Client, error) {
+	var caCerts []*x509.Certificate
+	// TODO: Integrate with TLS on metrics API


I assume we capture these TODOs in issues somewhere, correct?

Yes - we are currently tracking in ingest-dev, but will move to public issues shortly

thbkrkr

🚀 🧪 🍀

Co-authored-by: Peter Brachwitz <[email protected]>

robbavey · 2023-04-28T12:56:06Z

@barkbay @pebrc Nothing new to review, just wanted to get final approval from ECK team before merge

robbavey · 2023-04-28T13:35:11Z

@pebrc @barkbay @thbkrkr Thank you for the reviews, getting @kaisecheng and I to this point - from the Logstash side, I am happy to see this merged into main, but I don't have the authorization to do so... If one of you could please do the honours and press the button, please go ahead

robbavey · 2023-04-28T14:00:12Z

Also, are there any release notes/changelog entries that you would like me to add?

thbkrkr · 2023-04-28T14:03:34Z

Also, are there any release notes/changelog entries that you would like me to add?

This is handled separately and we will ask you at that time to contribute to it, thank you.

This adjusts the number of volumes expected from Beats sidecars in the Logstash Stack Monitoring unit tests. Why? Because we don't test PRs with an automatic merge of the main branch (🐛🐞), we missed that the tests in #6732 had to be updated to take into account the changes made by #6703, which adds a new temp volume to the Beats sidecars.

robbavey and others added 12 commits March 6, 2023 12:16

Merge remote-tracking branch 'upstream/main' into feature/logstash

450e61c

Merge remote-tracking branch 'upstream/main' into feature/logstash

a930555

Logstash controller unit and integration tests (#6575)

797f1c9

This commit adds extra unit tests for the Logstash Controller, including tests for config, controller, pods and services

Add Logstash Telemetry (#6562)

adca107

This commit adds information about Logstash to the set of ECK telemetry Co-authored-by: Thibault Richard <[email protected]>

Merge remote-tracking branch 'upstream/main' into feature/logstash

5811a64

Update all-crds.yml

169f79d

botelastic bot added the triage label Apr 26, 2023

barkbay added the >feature Adds or discusses adding a feature to the product label Apr 27, 2023

botelastic bot removed the triage label Apr 27, 2023

barkbay added triage v2.8.0 labels Apr 27, 2023

botelastic bot removed the triage label Apr 27, 2023

robbavey marked this pull request as ready for review April 27, 2023 13:32

flexitrev mentioned this pull request Apr 27, 2023

Logstash and Elastic Cloud on Kubernetes Announcement elastic/logstash#15030

Closed

barkbay approved these changes Apr 28, 2023

View reviewed changes

pebrc approved these changes Apr 28, 2023

View reviewed changes

thbkrkr approved these changes Apr 28, 2023

View reviewed changes

thbkrkr added the release-highlight Candidate for the ECK release highlight summary label Apr 28, 2023

Update pkg/apis/logstash/v1alpha1/logstash_types.go

ba98590

Co-authored-by: Peter Brachwitz <[email protected]>

robbavey added 2 commits April 28, 2023 08:46

Add comment to clusterName

c1930a2

Regenerate after comments

4186efb

pebrc merged commit b6b5075 into main Apr 28, 2023

thbkrkr mentioned this pull request Apr 28, 2023

Update Logstash Stack Monitoring unit tests #6741

Merged

thbkrkr deleted the feature/logstash branch May 10, 2023 11:50

thbkrkr added the :logstash label Jun 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce the Logstash operator for ECK #6732

Introduce the Logstash operator for ECK #6732

robbavey commented Apr 26, 2023 •

edited by thbkrkr

Loading

thbkrkr commented Apr 27, 2023 •

edited

Loading

robbavey commented Apr 27, 2023 •

edited by thbkrkr

Loading

robbavey commented Apr 27, 2023

robbavey commented Apr 27, 2023

barkbay left a comment

pebrc left a comment

pebrc Apr 28, 2023

pebrc Apr 28, 2023

pebrc Apr 28, 2023

robbavey Apr 28, 2023

thbkrkr left a comment

robbavey commented Apr 28, 2023

robbavey commented Apr 28, 2023

robbavey commented Apr 28, 2023

thbkrkr commented Apr 28, 2023

Introduce the Logstash operator for ECK #6732

Introduce the Logstash operator for ECK #6732

Conversation

robbavey commented Apr 26, 2023 • edited by thbkrkr Loading

thbkrkr commented Apr 27, 2023 • edited Loading

robbavey commented Apr 27, 2023 • edited by thbkrkr Loading

robbavey commented Apr 27, 2023

robbavey commented Apr 27, 2023

barkbay left a comment

Choose a reason for hiding this comment

pebrc left a comment

Choose a reason for hiding this comment

pebrc Apr 28, 2023

Choose a reason for hiding this comment

pebrc Apr 28, 2023

Choose a reason for hiding this comment

pebrc Apr 28, 2023

Choose a reason for hiding this comment

robbavey Apr 28, 2023

Choose a reason for hiding this comment

thbkrkr left a comment

Choose a reason for hiding this comment

robbavey commented Apr 28, 2023

robbavey commented Apr 28, 2023

robbavey commented Apr 28, 2023

thbkrkr commented Apr 28, 2023

robbavey commented Apr 26, 2023 •

edited by thbkrkr

Loading

thbkrkr commented Apr 27, 2023 •

edited

Loading

robbavey commented Apr 27, 2023 •

edited by thbkrkr

Loading