ANNOUNCEMENT: Handing over ongoing maintenance to community and contributors

[jmlrt]

When it comes to running the Elastic on Kubernetes infrastructure, we recommend Elastic Cloud on Kubernetes (ECK) as the best way to run and manage the Elastic Stack.

ECK offers many operational benefits for both our basic-tier and our enterprise-tier customers, such as spinning up cluster nodes that were lost on failed infrastructure, seamless upgrades, rolling cluster changes, and much much more.

With the release of the Elastic Stack Helm charts for Elastic version 8.5.1, we are handing over the ongoing maintenance of our Elastic Stack Helm charts to the community and contributors. The repository will finally be archived after 6 months time. Elastic Stacks deployed on Kubernetes through helm charts will still be fully supported under EOL limitations.

Since we want to provide an even better experience for our customers by running the Elastic Stack on Kubernetes, we will continue maintaining Helm charts applicable to ECK Custom Resources. These charts can be found in the ECK repository.

Helm charts will currently be maintained for ECK Enterprise-tier customers, however, we encourage the community to engage with the existing helm charts for the Elastic Stack and continue supporting their ongoing maintenance.

[darthf1]

Thank you for the announcement.

What is the advised upgrade path for migrating from this (ES) helm chart to the (ES) eck chart?

[brsolomon-deloitte]

What is the advised upgrade path for migrating from this (ES) helm chart to the (ES) eck chart?

The ability to do that is limited and this announcement is misleading in the first place.

First off, ECK is an operator and a set of CRDs. For example, it defines a custom resource called Elasticsearch. However, the Helm charts that support the ECK CRDs are minimalistic in nature and appear that they were thrown together in a rush. To pick an easy example, the eck-elasticsearch chart has no support or template for Ingress, which is a resource that would be expected in a Helm chart.

Second, some applications are not supported by ECK at all, one of them being Logstash, so a recommendation to "use ECK" in this case is pointless.

[ebuildy]

Also, a very important feature missing is we cannot use ECK with security best practice readOnlyRootFilesystem: true
(see elastic/cloud-on-k8s#6126) which is possible via Helm chart.

I think Helm charts should still leave, without new features but at least by supporting new releases.

[jmlrt]

What is the advised upgrade path for migrating from this (ES) helm chart to the (ES) eck chart?

It Depends.

There is not a single scenario that fits all cases for migrating from the Elasticsearch chart to ECK as there is no single scenario to migrate any Elasticsearch cluster to another one. A good start is ensuring that ECK has the features you need and is compatible with your the license you are using, then deploying ECK according to the doc. How you are migrating your data, it depends about your data size, availability requirements... It can be using Elasticsearch snapshot restored to ECK, cross-cluster replication or some other solutions. I'm not an expert in Elasticsearch or ECK myself, so I would advise contacting Elastic support if you are a customer.

[jmlrt]

For features missing in the ECK charts please open an issue in the cloud-on-k8s repo, so they can be tracked in the good place.

For cases that are not compatible with ECK, please note that the charts in this repo will not receive new features and new releases but can still be used with new Stack versions by overriding the imageTag value.

We also encourage the community to fork or take over the ownership of these repository. If you're interested by maintaining this repository, please reach us 🙏🏻.

[brsolomon-deloitte]

For features missing in the ECK charts please open an issue in the cloud-on-k8s repo, so they can be tracked in the good place.

Issues such as elastic/cloud-on-k8s#1453 have been open there for 3 years and running with no action. Deprecating this repo should have only been done once those issues were implemented and done.

We also encourage the community to fork or take over the ownership of these repository. If you're interested by maintaining this repository, please reach us 🙏🏻.

It would be a different story if Elastic stack was actually FOSS, but that excuse is long gone.

[Kushmaro]

@brsolomon-deloitte , not all issues opened can be acted upon at all times. We're doing our best to continue on working things that are of critical importance to our customers... please bear with us and expect some updates soon :)

[excalq]

Elastic's Logstash chart (among the others) is now being deprecated, in favor of ECK, which still does not include Logstash. This is most unfortunate, especially as we're paying Elastic Cloud customers with nothing provided for maintaining Logstash. This leaves us with even fewer resources.

[adopauco]

We just signed and paid a yearly contract for managed Elastic Cloud instance. We are using these charts to deploy filebeat and metricbeat to monitor our kubernetes infrastructure.

If I understand it correctly there is no way to use the ECK beats helm charts to send data to the managed elastic cloud. The only option for us is to rely on the community helm charts or to follow the official documentation and install filebeat using kubectl and manifest file with no simple way of upgrades / configuration / maintenance.
Is this really the case?
Did we just get left in the dust?

[Kushmaro]

@adopauco , there shouldn't be any issue for you to deploy beats using helm with ECK, and configure beats to send data to an Elastic Cloud deployment

[jcamu]

@adopauco , there shouldn't be any issue for you to deploy beats using helm with ECK, and configure beats to send data to an Elastic Cloud deployment

@Kushmaro , @jmlrt Is this the official recommandation ?
To use the cloud-on-k8s helm charts when we need to send k8s data with Beats to Elastic Cloud ?
Can we install beats Without the need to deploy eck-operator?
Or should we continue to use the helm-charts and hope that community will maintain it ? ...

Regarding licensing , what does it means for Elastic Cloud Customer ?

[JDKnobloch]

Is this supposed to be a community repo? Or a deprecated repo? @jmlrt @Kushmaro @excalq @ebuildy @darthf1

There have been roughly 10 community submitted PR's since this announcement 2 months ago, and you have not reviewed or even acknowledged in ANY way a SINGLE pull request.

What is going on? Why are customers being told to maintain your charts and ignored when they attempt to?
Y'all literally made this issue "Handing over ongoing maintenance to community and contributors", and IMMEDIATELY abandoned the repository and ignored any and all maintenance attempts from the community. What gives?

[Kushmaro]

@JDKnobloch thanks for the mention and pardon my late reply.
Such with any software project, there are finite resources that can attend a finite set of tasks in a given time. Even reviews, take up engineering time.
I'd be more than happy to see if we can attend to reviewing these community PRs soon, but will also ask you this -

would you like to be the maintainer of the charts?

[JDKnobloch]

No worries @Kushmaro, thanks for getting back.

I am not interested in maintaining, more just looking for a more finite definition regarding the repos current status, and I appreciate the explanation.

Given the current status, our team is electing to migrate our installations to the Bitnami maintained alternatives. This repo is ultimately a bit more convoluted and not as friendly for community maintenance in its current state (e.g. branches for diff versions that are outdated, in need of maintainers), so we've decided to take the plunge and migrate.

[ebuildy]

To be honest, we had a plat. license at Qwant, and use helm a lot . The end of helm chart has contributed (a little) to our decision to not renew it. Also, as I told, the operator is not production ready for us, as it's not possible to respect best security rules (especially read-only file system). Le mer. 18 janv. 2023, 12:44, JDKnobloch ***@***.***> a écrit :

…

Is this supposed to be a community repo? Or a deprecated repo? @jmlrt <https://github.com/jmlrt> @Kushmaro <https://github.com/Kushmaro> @excalq <https://github.com/excalq> @ebuildy <https://github.com/ebuildy> @darthf1 <https://github.com/darthf1> There have been roughly 10 community submitted PR's since this announcement *2 months ago*, and you *have not reviewed or even acknowledged in ANY way* a *SINGLE* pull request. What is going on? Why are *customers* being told to maintain your charts and *ignored* when they attempt to? Y'all literally made this issue "Handing over ongoing maintenance to community and contributors", and *IMMEDIATELY* abandoned the repository and ignored any and all maintenance attempts from the community. What gives? — Reply to this email directly, view it on GitHub <#1731 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJJZ2MMW6ENCWZ7IL74LHDWTATZPANCNFSM6AAAAAASCBEIHI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[RichiCoder1]

To use the cloud-on-k8s helm charts when we need to send k8s data with Beats to Elastic Cloud ?
Can we install beats Without the need to deploy eck-operator?
Or should we continue to use the helm-charts and hope that community will maintain it ? ...

Regarding licensing , what does it means for Elastic Cloud Customer ?

I'd like to echo this question in full and express my dissatisfaction. I'm interested in deploying Elastic Agent and Logstash to my k8s clusters. The former is ambigous as best as to both whether I'm allowed to use ECK via my subscription level as a managed customer and if it's the official recommendation, and the latter is wholly unsupported.

[Kushmaro]

@ebuildy i'm actually the PM for ECK myself. and ECK does support read-only filesystem (with a specific specification in the yaml file) we're aiming to make this default as well. ECK is definitely production ready.

I'll have that documented so its clearer.

[asazallesmilner]

Would really like some clarification on path forward for logstash. As it is not included in ECK that is not an option, nor can O even find the kubernetes yaml files i have used in the past for logstash in k8s.

[Kushmaro]

@asazallesmilner logstash will be available on ECK. Team is working on that.

[thbkrkr]

Until Logstash is fully managed by ECK, there is a recipe to demonstrate how to run Logstash on Kubernetes with Elasticsearch, Kibana and Filebeat deployed via ECK.

https://github.com/elastic/cloud-on-k8s/tree/main/config/recipes/logstash

[RichiCoder1]

It's great that it will be supported, but that also still leaves a question mark for Elastic Agent that would be awesome to have answers.

I also note that Pricing appears to now state that ECK is usable by Basic subscribers, no clarification anywhere I can find as to what falls into Basic and what falls into Enterprise.

[xeraa]

Yes, there's Basic. See https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-licensing.html#k8s-licensing which also links to the subscription page.

[asazallesmilner]

If you do have enterprise license, when logstash gets integrated into ECK will it count against your usage cap?

[flexitrev]

I'm happy to share that work Logstash on ECK is nearly complete - elastic/logstash#15030