Initial Commit of Basic Operator for Logstash #6404

robbavey · 2023-02-10T21:18:21Z

Initial commit of a simple operator.

The first operator will create:

A Service exposing the logstash metrics API, so it can be used for monitoring purposes
A secret holding logstash.yml
A StatefulSet deploying the logstash pods
- Pods deployed with a readiness probe testing against metrics API

The sample logstash yml file as located in config/samples/logstash/logstash.yaml will
create:

✗ kubectl tree logstash logstash-sample
NAMESPACE  NAME                                                  READY  REASON  AGE
default    Logstash/logstash-sample                              -              4m5s
default    ├─Secret/logstash-sample-ls-config                    -              4m4s
default    ├─Service/logstash-sample-ls-http                     -              4m5s
default    │ └─EndpointSlice/logstash-sample-ls-http-kwfsg       -              4m5s
default    └─StatefulSet/logstash-sample-ls                      -              4m4s
default      ├─ControllerRevision/logstash-sample-ls-79bd59f869  -              4m4s
default      ├─Pod/logstash-sample-ls-0                          True           3m59s
default      ├─Pod/logstash-sample-ls-1                          True           3m59s
default      └─Pod/logstash-sample-ls-2                          True           3m59s

And shows status:

✗ kubectl get logstash logstash-sample
NAME              AVAILABLE   EXPECTED   AGE    VERSION
logstash-sample   3           3          9m1s   8.6.1

This PR also includes basic tests:

Unit tests for names and validations in pkg/apis
End to end tests in e2e/logstash
- These test a single logstash, multiple logstashes and upgrade to latest 8.x
- Tests that logstash is setup and torn down correctly, and returns status correctly
Integration into TestSamples and TestVersionUpgradeOrdering in the test suites.

Still todo:

Unit tests for controller

Co-authored-by: Kaise Cheng [email protected]

robbavey · 2023-02-16T17:50:23Z

run/e2e-tests

robbavey · 2023-02-16T22:28:22Z

@thbkrkr, @kaisecheng, @naemono, @barkbay I think this is ready for review now.

There are a few things that are still in progress - the logstash controller still needs unit tests, but the basics are there, and I've added a unit tests for the api package, and added some basic end to end tests, and update the stack and sample tests to include logstash tests too - I haven't added to the Smoke tests, as from where I'm looking, they appear to be a somewhat limited and targeted set of tests. I've tested the logstash, smoke, stack and sample e2e tests locally, but would like to test the full e2e test suite too.

pebrc

Nice work! I left a few comments. But in general I think we could merge this already as a minimal viable version of Logstash support. The only thing that we should maybe decide is the API for the services/HTTP attribute in the CRD because that is really messy to change once we released it.

config/crds/v1/patches/kustomization.yaml

pebrc · 2023-02-18T21:19:33Z

pkg/apis/logstash/v1alpha1/logstash_types.go

+	//       a lot like `HTTPConfig`, but is applicable for more than just an HTTP endpoint, as logstash may need to
+	//       be opened up for other services: beats, TCP, UDP, etc, inputs
+	// +kubebuilder:validation:Optional
+	HTTP commonv1.HTTPConfig `json:"http,omitempty"`


Should we have it then right from the start as services? It is harder to change once we have release a version.

Yes. will change to []LogstashService.

pkg/apis/logstash/v1alpha1/name.go

pebrc · 2023-02-18T21:23:34Z

pkg/apis/logstash/v1alpha1/webhook.go

+		}
+
+		if len(errors) > 0 {
+			return apierrors.NewInvalid(groupKind, a.Name, errors)


I wonder why we return early here instead of letting the defaultChecks run.

you are right. It should go through all checks.

pebrc · 2023-02-18T21:24:32Z

pkg/controller/common/version/version.go

@@ -33,6 +33,7 @@ var (
 	// Due to bugfixes present in 7.14 that ECK depends on, this is the lowest version we support in Fleet mode.
 	SupportedFleetModeAgentVersions = MinMaxVersion{Min: MustParse("7.14.0-SNAPSHOT"), Max: From(8, 99, 99)}
 	SupportedMapsVersions           = MinMaxVersion{Min: From(7, 11, 0), Max: From(8, 99, 99)}
+	SupportedLogstashVersions       = MinMaxVersion{Min: From(8, 6, 0), Max: From(8, 99, 99)}


Is there a technical reason to support Logstash only as of version 8.6.0?

Logstash needs to add some features to work well with operator and we have no plan to backport those features to 7.17

pebrc · 2023-02-18T21:41:30Z

pkg/controller/logstash/logstash_controller.go

+	}
+
+	result, err := results.Aggregate()
+	k8s.EmitErrorEvent(r.recorder, err, logstash, events.EventReconciliationError, "Reconciliation error: %v", err)


We should rename this function to MaybeEmitErrorEvent as it ignores nil errors.

pebrc · 2023-02-18T21:48:46Z

pkg/controller/logstash/reconcile.go

+	defer tracing.Span(&params.Context)()
+	results := reconciler.NewResult(params.Context)
+
+	s, _ := sset.New(sset.Params{


error ignored. I think the error is actually always nil let's just remove the return value?

pebrc · 2023-02-18T21:50:11Z

pkg/controller/logstash/sset/sset.go

+	// store a hash of the sset resource in its labels for comparison purposes
+	sset.Labels = hash.SetTemplateHashLabel(sset.Labels, sset.Spec)
+
+	return sset, nil


error is always nil

pebrc · 2023-02-18T21:59:09Z

test/e2e/test/logstash/http_client.go

+	"github.com/elastic/cloud-on-k8s/v2/test/e2e/test"
+)
+
+type APIError struct {


Do we need the custom error type? I know we introduced something similar for Maps but it seems unnecessary there as well.

we don't need this

pebrc · 2023-02-18T22:26:52Z

pkg/controller/logstash/sset/sset.go

+			PodManagementPolicy:  appsv1.ParallelPodManagement,
+			RevisionHistoryLimit: params.RevisionHistoryLimit,
+			// build a headless service per StatefulSet, matching the StatefulSet labels
+			ServiceName: params.ServiceName,


The service referenced here is not a headless service but the HTTP service for Logstash which has a ClusterIP. This means the individual Logstash Pods are not resolvable via DNS (because that service always resolves to the ClusterIP of the service and the routing then happens with the iptables magic of kube proxy).

The K8s docs ask for a headless service for each stateful set. The thing I am not sure about is whether we need it if we don't need the ability to resolve the individual Logstash Pods via DNS.

I changed it to headless service following the k8s doc. The default service in Logstash is only for metrics collection. Having load balancing with iptables sounds more than it needs.

robbavey · 2023-02-27T19:46:29Z

pkg/apis/logstash/v1alpha1/name.go

+)
+
+const (
+	defaultServiceSuffix = "default"


What do you think about using the name "api" rather than "default", to reflect that this service is for the Logstash API? So this would show up as logstash-sample-ls-api, rather than logstash-sample-ls-default?

pebrc

I realise additional commits were pushed since you requested my review. Some of my comments might thus be outdated already.

pebrc · 2023-02-27T16:29:39Z

pkg/apis/logstash/v1alpha1/logstash_types.go

+	// +kubebuilder:validation:Optional
+	ConfigRef *commonv1.ConfigSource `json:"configRef,omitempty"`
+
+	// HTTP holds the HTTP layer configuration for the Logstash Metrics API


comment out of sync with code

pebrc · 2023-02-27T16:44:02Z

config/samples/logstash/logstash_svc.yaml

+  services:
+    - name: metrics
+      service:
+        spec:
+          type: ClusterIP
+          ports:
+            - port: 9600
+              name: "stats"
+              protocol: TCP
+              targetPort: 9600
+    - name: beats
+      service:
+        spec:
+          type: ClusterIP
+          ports:
+            - port: 5044
+              name: "beats"
+              protocol: TCP
+              targetPort: 5044


Could this not be a single service with two port declarations?

services: - name: ??? service: spec: type: ClusterIP ports: - port: 9600 name: "stats" protocol: TCP targetPort: 9600 - port: 5044 name: "beats" protocol: TCP targetPort: 5044

Is there still a use case for multiple services in Logstash e.g. one ClusterIP services and one with type LoadBalancer? I am not sure but maybe services plural does not hurt.

But I wonder if we need the naming?

services: - service: spec: type: ClusterIP ports: - port: 9600 name: "stats" protocol: TCP targetPort: 9600 - port: 5044 name: "beats" protocol: TCP targetPort: 5044

it does feel a little redundant, but if we removed the name from the spec, we'd need to define our own naming convention - would you suggest a simple numbered scheme for each service, eg logstash-sample-ls-0

I do think we may need the flexibility to define different service types, given the variety of plugins we support, and the use case to support multiple of those in a single logstash config.

That's a good point. I guess a ordinal system would be possible but there is also a metadata section in the service template that potentially allows users to specify a name (even though current docs say we will ignore it) so we have another redundancy. If it is easier to stick with the name attribute I think I would be OK with that too.

services: - service: metadata: name: "stats" spec: type: ClusterIP ports: - port: 9600 # N.B. https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/ name: "http-stats" protocol: TCP targetPort: 9600

On reflection, I think the name scheme works for us better than an ordinal scheme might do - it provides context when viewing service details, and also to allow the rules for the default API service to be overridden if necessary. I'd be happy to use the metadata method if that is more idiomatic, but I'd also like to be consistent with the rest of ECK wherever possible, and having a special case where metadata is meaningful seems like it might be a cause for confusion.

pebrc · 2023-02-27T16:48:52Z

pkg/apis/logstash/v1alpha1/logstash_types.go

+}
+
+type LogstashService struct {
+	Name string `json:"name,omitempty"`


Missing comment. But I left a comment regarding the API up top at the sample to remove this all together.

pkg/controller/logstash/service.go

pebrc · 2023-02-27T19:47:39Z

pkg/controller/logstash/reconcile.go

+	s := sset.New(sset.Params{
+		Name:                 logstashv1alpha1.Name(params.Logstash.Name),
+		Namespace:            params.Logstash.Namespace,
+		ServiceName:          logstashv1alpha1.DefaultServiceName(params.Logstash.Name),


You are setting the default service name here but you are only creating the default service if no custom service is defined.

pebrc · 2023-02-27T19:51:17Z

pkg/apis/logstash/v1alpha1/logstash_types.go

+	// Service defines the template for the associated Kubernetes Service object.
+	Service commonv1.ServiceTemplate `json:"service,omitempty"`
+	// TLS defines options for configuring TLS for HTTP.
+	TLS commonv1.TLSOptions `json:"tls,omitempty"`


This seems unused? Is there a plan to implement certificate management for Logstash?

Yes, we would like to add support for TLS for the metrics API in a follow-up commit.

pebrc · 2023-02-27T19:53:01Z

pkg/controller/logstash/driver.go

+	defer tracing.Span(&params.Context)()
+	results := reconciler.NewResult(params.Context)
+
+	_, err := reconcileServices(params)


You are ignoring the return value here. Can it simply be removed from the function?

It is for a follow-up commit for TLS support

@kaisecheng or it could be removed and added in the upcoming PR that adds TLS support? Or at least comment that section to let reviewers know what it is for?

We will add TLS support in the near future, so prefer to keep it. I can add a TODO to explain

robbavey · 2023-02-27T21:48:35Z

pkg/controller/logstash/service.go

+// If api.http.port is customized, user is expected to config Services.
+// When Services exist, the port 9600 does not attach to any of Service.
+func reconcileServices(params Params) ([]corev1.Service, error) {
+	if len(params.Logstash.Spec.Services) == 0 {


Rather than checking if the number of services is 0, we should probably check whether a service has been spec'd with the same name as the default service - I think it may make sense to rename this to api, or internal, rather than default, WDYT?

we can make a default headless service regardless of user input similar to ES

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE elasticsearch-sample-es-default ClusterIP None <none> 9200/TCP 17h elasticsearch-sample-es-http ClusterIP 10.98.229.240 <none> 9200/TCP 17h elasticsearch-sample-es-internal-http ClusterIP 10.96.255.68 <none> 9200/TCP 17h elasticsearch-sample-es-transport ClusterIP None <none> 9300/TCP 17h

For user defined service, name it with suffixapi. If user doesn't provide input, create a ClusterIP service with suffix http for metrics.

I was wondering if we could call the service for metrics api,internal or http, regardless of whether they have set values in the spec. And if they define a service with the same name, it overrides the default settings, allowing the user to change the cluster type from the default if they need to - say if they have an external monitoring tool that reads from the logstash API.

yes. Using the name as an identifier to override default settings (eg. port) sounds good.

My concern is this comment

K8s docs ask for a headless service for each stateful set

Regardless of user's setting in spec, I think we can always create headless service for Logstash

Yes, I agree - I do think we need to always create a headless service

Initial commit of a simple operator. The first operator will create: * A Service exposing the logstash metrics API, so it can be used for monitoring purposes * Secrets holding logstash.yml * A StatefulSet deploying the logstash pods * Pods with default liveness and readiness probes The sample logstash yml file as located in config/samples/logstash/logstash.yaml will create: ``` ✗ kubectl tree logstash logstash-sample NAMESPACE NAME READY REASON AGE default Logstash/logstash-sample - 4m5s default ├─Secret/logstash-sample-ls-config - 4m4s default ├─Service/logstash-sample-ls-http - 4m5s default │ └─EndpointSlice/logstash-sample-ls-http-kwfsg - 4m5s default └─StatefulSet/logstash-sample-ls - 4m4s default ├─ControllerRevision/logstash-sample-ls-79bd59f869 - 4m4s default ├─Pod/logstash-sample-ls-0 True 3m59s default ├─Pod/logstash-sample-ls-1 True 3m59s default └─Pod/logstash-sample-ls-2 True 3m59s ``` And shows status: ``` ✗ kubectl get logstash logstash-sample NAME AVAILABLE EXPECTED AGE VERSION logstash-sample 3 3 9m1s 8.6.1 ``` Still TODO: * Unit Tests * End to end Tests * Certificate handling on the HTTP Metrics API Tidy up Co-authored-by: Kaise Cheng <[email protected]>

`

Rudimentary tests for validation and naming

Update service name verification to `default` from `http`

improve comment Co-authored-by: Peter Brachwitz <[email protected]>

This commit adds support to enable users to specify a `default` service to override the provided defaults. This commit also ensures that the `default` service is added if other services are defined

The default service represents the logstash API, so this commit updates the service name and associated methods to reflect that. This should also be more intuitive for users who wish to change the settings for the service.

robbavey · 2023-03-02T03:11:14Z

@pebrc This should be ready for another round. Thanks!

barkbay

Do we also want to update:

The memory aggregator:

cloud-on-k8s/pkg/license/aggregator.go

Lines 38 to 56 in 48d6eed

    
           // AggregateMemory aggregates the total memory of all Elastic managed components 
        
           func (a Aggregator) AggregateMemory(ctx context.Context) (resource.Quantity, error) { 
        
           	var totalMemory resource.Quantity 
        
           	for _, f := range []aggregate{ 
        
           		a.aggregateElasticsearchMemory, 
        
           		a.aggregateKibanaMemory, 
        
           		a.aggregateApmServerMemory, 
        
           		a.aggregateEnterpriseSearchMemory, 
        
           	} { 
        
           		memory, err := f(ctx) 
        
           		if err != nil { 
        
           			return resource.Quantity{}, err 
        
           		} 
        
           		totalMemory.Add(memory) 
        
           	} 
        
           	return totalMemory, nil 
        
           }

The telemetry reporter:

cloud-on-k8s/pkg/telemetry/telemetry.go

Lines 112 to 135 in 48d6eed

    
           func (r *Reporter) getResourceStats(ctx context.Context) (map[string]interface{}, error) { 
        
           	span, _ := apm.StartSpan(ctx, "get_resource_stats", tracing.SpanTypeApp) 
        
           	defer span.End() 
        
           	stats := map[string]interface{}{} 
        
           	for _, f := range []getStatsFn{ 
        
           		esStats, 
        
           		kbStats, 
        
           		apmStats, 
        
           		beatStats, 
        
           		entStats, 
        
           		agentStats, 
        
           		mapsStats, 
        
           		scpStats, 
        
           	} { 
        
           		key, statsPart, err := f(r.client, r.managedNamespaces) 
        
           		if err != nil { 
        
           			return nil, err 
        
           		} 
        
           		stats[key] = statsPart 
        
           	} 
        
           	return stats, nil 
        
           }

?

pkg/apis/logstash/v1alpha1/webhook.go

pkg/apis/logstash/v1alpha1/name.go

pkg/controller/logstash/reconcile.go

pkg/controller/logstash/service.go

barkbay · 2023-03-02T10:38:15Z

pkg/controller/logstash/sset/sset.go

+			UpdateStrategy: appsv1.StatefulSetUpdateStrategy{
+				Type: appsv1.RollingUpdateStatefulSetStrategyType,
+			},
+			// we don't care much about pods creation ordering, and manage deletion ordering ourselves,


[...] and manage deletion ordering ourselves

IIUC deletion is managed by the sts controller (which raises the question of whether we want to expose RollingUpdateStatefulSetStrategy, can be discussed in a separate issue/PR)

I agree - I think we will want to expose this at a later date

barkbay · 2023-03-02T10:58:56Z

pkg/apis/logstash/v1alpha1/logstash_types.go

+)
+
+// LogstashSpec defines the desired state of Logstash
+type LogstashSpec struct {


Is there a plan to add a reference to an Elasticsearch resource?

Yes, there will be a follow up PR to add support for Elasticsearch references

barkbay · 2023-03-02T11:00:00Z

config/samples/logstash/logstash.yaml

+apiVersion: elasticsearch.k8s.elastic.co/v1
+kind: Elasticsearch
+metadata:
+  name: elasticsearch-sample


The purpose of this Elasticsearch resource is not clear.

Will remove

barkbay · 2023-03-02T12:19:40Z

hack/operatorhub/config.yaml

@@ -30,6 +30,9 @@ crds:
  - name: stackconfigpolicies.stackconfigpolicy.k8s.elastic.co
    displayName: Elastic Stack Config Policy
    description: Elastic Stack Config Policy
+  - name: logstashes.logstash.k8s.elastic.co


We may also want to update hack/operatorhub/templates/csv.tpl

Co-authored-by: Michael Morello <[email protected]>

robbavey · 2023-03-02T15:24:45Z

Do we also want to update:

The memory aggregator:

cloud-on-k8s/pkg/license/aggregator.go

Lines 38 to 56 in 48d6eed

// AggregateMemory aggregates the total memory of all Elastic managed components

func (a Aggregator) AggregateMemory(ctx context.Context) (resource.Quantity, error) {

var totalMemory resource.Quantity

for _, f := range []aggregate{

a.aggregateElasticsearchMemory,

a.aggregateKibanaMemory,

a.aggregateApmServerMemory,

a.aggregateEnterpriseSearchMemory,

} {

memory, err := f(ctx)

if err != nil {

return resource.Quantity{}, err

}

totalMemory.Add(memory)

}

return totalMemory, nil

}

@barkbay Good question - does this value factor into licensing?

I think this might be a good candidate for a follow-on PR, once we have a definitive answer on how Logstash fits into licensing in general.

cc @flexitrev

The telemetry reporter:

cloud-on-k8s/pkg/telemetry/telemetry.go

Lines 112 to 135 in 48d6eed

func (r *Reporter) getResourceStats(ctx context.Context) (map[string]interface{}, error) {

span, _ := apm.StartSpan(ctx, "get_resource_stats", tracing.SpanTypeApp)

defer span.End()

stats := map[string]interface{}{}

for _, f := range []getStatsFn{

esStats,

kbStats,

apmStats,

beatStats,

entStats,

agentStats,

mapsStats,

scpStats,

} {

key, statsPart, err := f(r.client, r.managedNamespaces)

if err != nil {

return nil, err

}

stats[key] = statsPart

}

return stats, nil

}

?

Yes - we have plans to add Telemetry for the logstash operator in a follow up PR.

robbavey · 2023-03-02T17:21:10Z

@barkbay Thanks for the comments - I think I addressed the points from your review

barkbay

Overall LGTM, a few other nits:

Do we plan to add some unit tests? There is none at the moment in the controller package. I feel like it would make sense for functions like reconcileServices or buildPodTemplate.
Other ECK resources have a Health field in their status. I was wondering if it would make sense to add one for Logstash. The status already holds the number of availableNodes, I don't know if we could offer more to help the user understand the Logstash resource health?
Should we already open an issue for each feature/task that it is still to be done or discussed? A few that I have in mind:
- Adding the Elasticsearch reference
- Exposing RollingUpdateStatefulSetStrategy
- Implementing TLS
- Update the documentation
- Discussing telemetry and usage report

robbavey · 2023-03-06T14:42:20Z

@barkbay Unit tests is a good point - I'll add. And I'll take a look at the health and see what makes sense for Logstash

As for follow up tasks - we have a plan to use this PR as a building block for additional functionality. The additional functionality includes stack monitoring, better support for handling logstash pipelines, Elasticsearch references, TLS support for monitoring, etc.

See https://github.com/elastic/ingest-dev/issues/1441 for meta issue

robbavey · 2023-03-06T15:49:51Z

@barkbay - given the LGTM, would you have any objections to having this committed, and adding unit tests later today/early tomorrow (especially as this PR is targeted to a feature/logstash branch) - I would like to unblock the in progress stack monitoring PR

barkbay

LGTM!

botelastic bot added the triage label Feb 10, 2023

robbavey changed the title ~~Initial Commit of Basic Operator for Logstash~~ Initial Commit of Basic Operator for Logstash [wip] Feb 10, 2023

thbkrkr added the >feature Adds or discusses adding a feature to the product label Feb 13, 2023

botelastic bot removed the triage label Feb 13, 2023

robbavey force-pushed the basic_operator branch 7 times, most recently from ceaf097 to c9e8bab Compare February 16, 2023 17:48

robbavey force-pushed the basic_operator branch from c9e8bab to 5713bb4 Compare February 16, 2023 18:08

robbavey marked this pull request as draft February 16, 2023 22:23

robbavey changed the title ~~Initial Commit of Basic Operator for Logstash [wip]~~ Initial Commit of Basic Operator for Logstash Feb 16, 2023

robbavey marked this pull request as ready for review February 16, 2023 22:24

robbavey requested a review from thbkrkr February 16, 2023 23:15

pebrc reviewed Feb 18, 2023

View reviewed changes

kaisecheng mentioned this pull request Feb 21, 2023

Stack monitoring for Logstash #6436

Merged

kaisecheng requested a review from pebrc February 24, 2023 15:52

robbavey commented Feb 27, 2023

View reviewed changes

pebrc reviewed Feb 27, 2023

View reviewed changes

robbavey commented Feb 27, 2023

View reviewed changes

robbavey and others added 6 commits February 28, 2023 18:16

Comment out certs for HTTPS for now

02a5451

Fix up linter issus

b72182f

Generate API docs

9cf0a45

`

First set of unit tests

4d62ab5

Rudimentary tests for validation and naming

Fix goimports

cf1379c

kaisecheng and others added 10 commits February 28, 2023 18:16

change the default Service to headless service

0a37afe

fix service test

df96068

fix podTemplate and doc generation

4c414bd

Fix Service tests

018dfbc

Update service name verification to `default` from `http`

Update comments to reflect change from HTTPConfig to LogstashService

de1fb22

Update pkg/controller/logstash/service.go

41612f7

improve comment Co-authored-by: Peter Brachwitz <[email protected]>

remove duplicate err check

a0d176a

Add support for overriding "default" service

b9610da

This commit adds support to enable users to specify a `default` service to override the provided defaults. This commit also ensures that the `default` service is added if other services are defined

Replace 'default' service name with 'api'

0c9409a

The default service represents the logstash API, so this commit updates the service name and associated methods to reflect that. This should also be more intuitive for users who wish to change the settings for the service.

merge from feature/logstash

6167cb7

robbavey force-pushed the basic_operator branch from 1a06bb6 to 6167cb7 Compare February 28, 2023 23:27

Add tests and checks for services and endpoints

d4c5e05

robbavey force-pushed the basic_operator branch from 04ac558 to d4c5e05 Compare March 2, 2023 00:35

robbavey requested a review from pebrc March 2, 2023 03:10

barkbay self-requested a review March 2, 2023 08:47

barkbay reviewed Mar 2, 2023

View reviewed changes

pkg/apis/logstash/v1alpha1/webhook.go Outdated Show resolved Hide resolved

pkg/apis/logstash/v1alpha1/name.go Outdated Show resolved Hide resolved

pkg/controller/logstash/reconcile.go Outdated Show resolved Hide resolved

pkg/controller/logstash/service.go Outdated Show resolved Hide resolved

barkbay reviewed Mar 2, 2023

View reviewed changes

robbavey and others added 2 commits March 2, 2023 09:08

Update pkg/apis/logstash/v1alpha1/webhook.go

1f77430

Co-authored-by: Michael Morello <[email protected]>

Update pkg/controller/logstash/service.go

b677c65

Co-authored-by: Michael Morello <[email protected]>

Addressing code review comments

a797940

barkbay reviewed Mar 6, 2023

View reviewed changes

barkbay approved these changes Mar 6, 2023

View reviewed changes

robbavey merged commit 47beb32 into elastic:feature/logstash Mar 6, 2023

thbkrkr added the :logstash label Jun 23, 2023

	// AggregateMemory aggregates the total memory of all Elastic managed components
	func (a Aggregator) AggregateMemory(ctx context.Context) (resource.Quantity, error) {
	var totalMemory resource.Quantity

	for _, f := range []aggregate{
	a.aggregateElasticsearchMemory,
	a.aggregateKibanaMemory,
	a.aggregateApmServerMemory,
	a.aggregateEnterpriseSearchMemory,
	} {
	memory, err := f(ctx)
	if err != nil {
	return resource.Quantity{}, err
	}
	totalMemory.Add(memory)
	}

	return totalMemory, nil
	}

	func (r *Reporter) getResourceStats(ctx context.Context) (map[string]interface{}, error) {
	span, _ := apm.StartSpan(ctx, "get_resource_stats", tracing.SpanTypeApp)
	defer span.End()

	stats := map[string]interface{}{}
	for _, f := range []getStatsFn{
	esStats,
	kbStats,
	apmStats,
	beatStats,
	entStats,
	agentStats,
	mapsStats,
	scpStats,
	} {
	key, statsPart, err := f(r.client, r.managedNamespaces)
	if err != nil {
	return nil, err
	}
	stats[key] = statsPart
	}

	return stats, nil
	}

Initial Commit of Basic Operator for Logstash #6404

Initial Commit of Basic Operator for Logstash #6404

Conversation

robbavey commented Feb 10, 2023 • edited Loading

robbavey commented Feb 16, 2023

robbavey commented Feb 16, 2023

pebrc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaisecheng Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pebrc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey Feb 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roaksoax Mar 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey commented Mar 2, 2023

barkbay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey commented Mar 2, 2023

robbavey commented Mar 2, 2023

barkbay left a comment

Choose a reason for hiding this comment

robbavey commented Mar 6, 2023

robbavey commented Mar 6, 2023

barkbay left a comment

Choose a reason for hiding this comment

robbavey commented Feb 10, 2023 •

edited

Loading

kaisecheng Feb 24, 2023 •

edited

Loading

robbavey Feb 27, 2023 •

edited

Loading

roaksoax Mar 1, 2023 •

edited

Loading