Module update and path to supporting 7.x

[fatmcgav]

Hey all,

So I wanted to open this issue to give anyone watching an update on what's next for this module.

Firstly, apologies from Elastic that this module has not been given the time and effort it deserves.

The plan for the next steps for this module are as follows:

• Refresh dependencies and supported versions.

  This will unfortunately require dropping support for any Puppet versions older than 5.x, and any Elasticsearch version older than 6.x.

• Remove support for installing and managing multiple instances on a single host.

  This was the "recommended" method a few years ago in order to maximise the memory utilisation for a given host containing more than 32GB of RAM. However other methods have become more prevalent since, such as running Elasticsearch in Docker.
  Therefore in order to massively simplify this module we will be removing support for installing and managing multiple instances on a single host.

• Update the module to support installing Elasticsearch 7.

  This is the biggest thing that most of you are asking for - Support for installing and managing Elasticsearch 7.x.
  The good news is: It's coming!

Feel free to ask any questions or raise any concerns you might have with any of the above, and we'll do our best to work with you.

And thanks again to everyone who's taken the time to open issues or PR's - It's really appreciated.

[faxm0dem]

We're using both ansible and puppet for managing our infrastructure.
It would make sense to have both puppet and ansible modules to generate "similar" configuration state on the hosts, so that minimal conflict might happen when using both configuration management tools, or when switching from one to the other.

[zofog]

Please ensure support for upgrading from 6.x to 7.x for instances that were installed using this module.

[andrewbierbaum]

Since Ubuntu 20.04 is near and there are starting to be documtation for 8.x, please test at least some against if possible against those so the transition going forward is as easy as possible.

Also, version 7.x can be run on this module if jvm setting are set directly. It's not clear to me why those jvm settings are not just pulled more directly from the base installation with the ability to tweak them from there. Ideally this module would pull as much as possible from what's known to be a good base installation and provide the ability to change the important configurations from there.

Thanks for your work on this!

[cdenneen]

@fatmcgav I honestly disagree with the multiple instances removal for this module. The docker options while nice should be handled by other methods like docker-compose or helm charts for k8s. This is still a “puppet” module for deploying and maintaining elastic on a full VM/bare metal install. It definitely has different audience than the “docker” folks. Unless the plan for this module is to use docker module as dependency and spin up elastic inside containers leveraging ECE or something I feel that instance support should remain for the foreseeable future. The users of this module most likely wouldn’t even be using it, or puppet for that matter, in the case of deploying to a container setup. With that said I don’t think that allows for a “one size fits all model” unless, as mentioned above, this module will now support the installation of elastic on containers via puppet in which multi instance would still be supported but via containers rather than systemd/jvm processes.

[andrewbierbaum]

Master installed on the same host as a data node should absolutely be removed. That is already covered by having both node.master and node.data as true. This is a better solution than having two instances of elasticsearch running on the same VM.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html

Having two separate instances of elastic running was sited by the previous maintainer as being on of the main challenges of moving to his forward. It should be removed.

I'm also using both puppet and docker images for our deploy. I think most shops working with an updated puppet and elastic will have some docker workflow.

[cdenneen]

Running data and master in same node is great for non-prod use. However in a production level having the master and data on separate JVMs to avoid ingestion OOM exceptions taking down the master instance is always best. In a VM world this is fine but in real steel hardware can have tons of memory so to utilize hardware with 64gb+ memory best is to run multiple instances to utilize the memory properly.

[cdenneen]

I'm also using both puppet and docker images for our deploy. I think most shops working with an updated puppet and elastic will have some docker workflow.

I think that is a pretty big assumption. If you are using containers there is too much bloat in this module to maintain that deployment. You have no reason for half of what this controls. Unless your running full blown systemd containers not sure how this module would manage elastic on docker.
Moving towards containers I don’t disagree with but since this module wouldn’t be able to maintain it, it’s for a different audience than real steel or vm deployments.

[mookie-]

At first, thank you very much for providing a puppet module for elasticsearch. Today I've noticed this issue and I've read this:

Remove support for installing and managing multiple instances on a single host.

This was the "recommended" method a few years ago in order to maximise the memory utilisation for a given host containing more than 32GB of RAM. However other methods have become more prevalent since, such as running Elasticsearch in Docker.
Therefore in order to massively simplify this module we will be removing support for installing and managing multiple instances on a single host.

I don't understand it. If you use elasticsearch in docker, then you probably don't use this puppet module. I don't want to have 5 dockers, I want 5 elasticsearch instances on my machine. Also to massively simplify this module, the first steps from my point of view would be to make it more "puppet" like. You have a lot of custom functions/types/... in it which are not needed you could have done that with puppet standard features.
Why did you build a template engine in your module if there already is a template engine in puppet? I remember debugging this module multiple times and getting really frustrated due to this. This also makes it harder to contribute to this module.

[fatmcgav]

Firstly, thank you to everyone who has taken the time to provide feedback, 👍's etc, and apologies for not updating sooner.

In the background, we've been working on resolving some latent issues with the failing tests, updating dependencies etc.

Please be assured that we don't make the decision to remove the multi-instance functionality lightly.
However running multiple instances natively on a single node has often led to unpredictable performance profiles, unexpected behaviour, etc and therefore is no longer an officially supported deployment method.
It also results in much more complexity within this module, which makes adding support for new versions etc increasingly complicated and time-consuming.

With that being said, we appreciate that in certain scenarios, such as dev or test clusters etc, you are often forced to work with the hardware that's available. So we will be exploring whether we can provide some more guidance, examples etc around running multiple instances on a single node using other tools, such as Docker.

I'll follow up and address more specific feedback in separate comments.

[fatmcgav]

Please ensure support for upgrading from 6.x to 7.x for instances that were installed using this module.

@zofog We will be working to make the upgrade and migration path between 6.x and 7.x as painless as possible. However it's possible that there may not be a simple migration path. Further details will be provided when possible.

Since Ubuntu 20.04 is near and there are starting to be documtation for 8.x, please test at least some against if possible against those so the transition going forward is as easy as possible.

Also, version 7.x can be run on this module if jvm setting are set directly. It's not clear to me why those jvm settings are not just pulled more directly from the base installation with the ability to tweak them from there. Ideally this module would pull as much as possible from what's known to be a good base installation and provide the ability to change the important configurations from there.

Thanks for your work on this!

@andrewbierbaum Matching as much as possible the "default" JVM options etc will be part of this work, as we are aware that the divergence is an issue.
We'll review adding Ubuntu 20.04 (Focal) to the support matrix at the earliest possibility.

@cdenneen I appreciate that when running on bare metal, you will want to maximise the resource utilisation of the underlying hardware. However that is where Docker provides better guarantees around resource separation etc.
Whilst this module won't directly support provisioning multiple instances via Docker, we will be looking at providing further guidance to support this usage model in the near future.

I don't understand it. If you use elasticsearch in docker, then you probably don't use this puppet module. I don't want to have 5 dockers, I want 5 elasticsearch instances on my machine. Also to massively simplify this module, the first steps from my point of view would be to make it more "puppet" like. You have a lot of custom functions/types/... in it which are not needed you could have done that with puppet standard features.
Why did you build a template engine in your module if there already is a template engine in puppet? I remember debugging this module multiple times and getting really frustrated due to this. This also makes it harder to contribute to this module.

@mookie- It's worth noting that Puppet itself has changed a lot over recent years, with the introduction of richer typing etc. And the language is continually evolving and adding new functionality. So reviewing the existing "custom" pieces is part of the plan.

[cdenneen]

@cdenneen I appreciate that when running on bare metal, you will want to maximise the resource utilisation of the underlying hardware. However that is where Docker provides better guarantees around resource separation etc.
Whilst this module won't directly support provisioning multiple instances via Docker, we will be looking at providing further guidance to support this usage model in the near future.

I don't understand it. If you use elasticsearch in docker, then you probably don't use this puppet module. I don't want to have 5 dockers, I want 5 elasticsearch instances on my machine. Also to massively simplify this module, the first steps from my point of view would be to make it more "puppet" like. You have a lot of custom functions/types/... in it which are not needed you could have done that with puppet standard features.
Why did you build a template engine in your module if there already is a template engine in puppet? I remember debugging this module multiple times and getting really frustrated due to this. This also makes it harder to contribute to this module.
@fatmcgav The issue here is using docker as you said is outside the scope of this module and in that case you wouldn't honestly be using a puppet module but more of something like a helm chart. So since this module is geared more towards VM's or bare metal having the option to maintain multi-instance I think still makes sense.
If this module wants to completely revamp and handle ECE or something that will leverage containers under the covers then that handles the multi-instance model as well.

I honestly think if you diverge from being able to support multi-instance I see people moving away from this module completely in favor of ECK or Elastic's official Helm Chart. Not to say there wouldn't be some way to cobble this together in a container scenario but honestly if you are setting things up in a "container" fashion... you aren't going to use something that's basically shoe horning a "full stack" solution into a "container" model.

[gvenaas]

I am wondering about the use of the instance notation that now is like this:

elasticsearch::instance { 'es-01':
}

Are You planning to remove that when dealing with the multi instance part of the module?

If so, do You have any examples of how we have to deal with changing from the old to the new notation?

[fatmcgav]

Are You planning to remove that when dealing with the multi instance part of the module?

If so, do You have any examples of how we have to deal with changing from the old to the new notation?

@gvenaas yes, the intention is to remove the elasticsearch::instance defined type.

With regards to migration; not currently. However we will be working on making the migration as simple as possible.

[packetrevolt]

Can removing multi-instance be reconsidered?

Removing multi-instance support and telling users to re-deploy an Elasticsearch cluster using Docker is a pretty large request. Especially the types of users that would go through the effort to use Puppet to deploy multi-instances; often times they would be the users with many hundreds of nodes per cluster and many clusters.

Adding to what @cdenneen said above. Once you get big enough, you often run Elasticsearch bare metal and you use some heavy automation. VM's are way too expensive. Docker is an unnecessary complexity unless you are already a docker only shop. Your servers will have way more hardware then any single instance of Elasticsearch would be able to use on its own. And you have so many servers and nodes that it is impossible to manage through any other way then automation (like this Puppet module). What is likely to happen is many of the large users of this Puppet module will either hard fork it or stop using it all together since the official stance from Elastic is to use Docker (and thus...not Puppet) if you have bare metal hardware.

[haghayeghh]

any update for ver7 support ?

[jbillen]

any update for ver7 support ?

+1

[fatmcgav]

Hey there,

Firstly, apologies for the radio silence on this issue.

The good news is that the master branch now supports deploying Elasticsearch 7.x!!!
So if you feel up for it, you can clone the master branch and take things for a spin.

The caveat to that is that the documentation hasn't been completely updated yet. We also don't have a supported upgrade path from 6.x to 7.x yet, due to the breaking changes around removing the multi-instance support which has changed all the paths on disk etc.
So if you did want to try master, it would be best to only target new deployments rather than trying to update an existing one.

The documentation updates and upgrade path should be coming in the near future.

If anyone does try the new functionality, feedback is most definitely welcome.

Thanks again for all your patience.

[smortex]

If anyone does try the new functionality, feedback is most definitely welcome.

@fatmcgav I just did a quick deployment on Debian 10 (nothing fancy, just a quick "first run" in a VM to give me an idea and for the purpose of evaluating the changes to the module), and everything was smooth. Better: everything was as I expected: no need to re-apply a catalog to "fix" the configuration after an update ; no need to set dubious env vars to actually update elasticsearch ❤️ 🧡 💛 💚 💙

Two things I though while testing:

1. The dependency on elastic/elastic_stack in metadata.json could be adjusted (< 7.0.0 → < 8.0.0);
2. A few PR could be closed now that 7.x support is in the master branch (elastic 7.x support #1061, elastic 7.x support (originally from uberjew666) #1082): I was subscribed to all of them but not to this issue and I only discovered it today.

[grant-veepshosting]

Trying to get Version 7 installed from the master branch per @fatmcgav advice above, using this super simple config -
contain ::java contain ::elastic_stack::repo class { 'elasticsearch': autoupgrade => true, config => {'network.host' => '127.0.0.1'}, jvm_options => ['-Xms256m','-Xmx1280m'], restart_on_change => true, ssl => false, }

I get -
Error: Could not autoload puppet/provider/elasticsearch_user_file/oss_xpack: undefined method ``oss_xpack_config' for Puppet::Type::Elasticsearch_user_file::ProviderOss_xpack:Class Error: Could not autoload puppet/type/elasticsearch_user_file: Could not autoload puppet/provider/elasticsearch_user_file/oss_xpack: undefined method ``oss_xpack_config' for Puppet::Type::Elasticsearch_user_file::ProviderOss_xpack:Class Error: Evaluation Error: Error while evaluating a Virtual Query, Could not autoload puppet/type/elasticsearch_user_file: Could not autoload puppet/provider/elasticsearch_user_file/oss_xpack: undefined method ``oss_xpack_config' for Puppet::Type::Elasticsearch_user_file::ProviderOss_xpack:Class (file: /etc/puppetlabs/code/environments/production/modules/elasticsearch/manifests/init.pp, line: 605, column: 6) on node <node>

Any ideas?

[grant-veepshosting]

EDIT - removing or renaming the file 'lib/puppet/provider/elastic_parsedfile.rb' has resolved the above issue, but not likely a long term solution.

[grant-veepshosting]

UPDATE - I think the above condition is resolved by upgrading the "elastic_stack" module to the latest Forge v7.0.0, in case anyone else encounters it.

[grant-veepshosting]

From internal testing, it looks like mandatory/required files: ['jvm.options','log4j2.properties'] are not currently managed by the module. Although they're in the directory when the default installer runs, when transitioning clients from a legacy version of the module (eg, last Forge release v6.4.0) these files are missing by design of the previous version of the module.

[StanvanHoorn]

Any updates on when this is ready to be released?

[mikeycarlton]

What's the current timeline look like on getting the 7.x support added? We are definitely patiently waiting. :)

[Antiarchitect]

@fatmcgav We're waiting too.

[gvenaas]

And waiting as well

[fatmcgav]

Apologies again for the lack of progress here... Other priorities took over...

With that being said, I'd like to announce that version 7.0.0 of the module has just been published ot the Puppet Forge.
As always, as this is a major version bump with a number of breaking changes, please PLEASE test this before using it in production.

I'd also like to call out the Warning from the README - This module DOES NOT support upgrading or migrating existing multi-instance deployments!
I say again, if you run this against an existing installation, it will most likely BREAK THINGS!!!

Our current recommendation is to deploy Elasticsearch to new nodes and migrate the data over.
You might be able to deploy Elasticsearch using the new version of this module and point it at an existing data directory; however that hasn't been tested as yet by us.

We're planning on trying to make this story better, however I'm honestly not sure when that will happen.

Feel free to reach out with any issues or questions, and thanks again for everyone's patience and comments on this issue.

Hope everyone has a good festive period and the computer gods are kind!

[tamama]

Agreed.

1. Elasticsearch 7.x is not backward compatible with 6.x.
2. Do not play with fire, unless you are willing to lose production data.
3. Worst-case scenario, do backup. But... my experience tells me that recovery will take for ages.

Recommendation: Deploy to new nodes that are fresh.

Even the rolling upgrade by official ELK does not provide rolling downgrade. So, when you run into trouble during installation, good luck to you :(

[rlewis-droid]

i think adding an explicit variable for node role would help, node.master and node.data, etc
unless you recommend just hashing this in through config hash

[gvenaas]

Any updates on the "upgrade path" challenge.
Since Puppet don't allow two versions of the module within an environment, its troublesome to find a solution today.

How about allowing a module rename. Having one for Elasticsearch6, one for Elasticsearch7 etc. Then we could use them in parallel within the same puppet environment.

[Betriebsrat]

Hi,
we are still running into the following errors:

Error: Could not autoload puppet/provider/elasticsearch_role/shield: undefined method `shield_config' for Puppet::Type::Elasticsearch_role::ProviderShield:Class
Error: Failed to load custom type 'elasticsearch_role' from '/etc/puppetlabs/code/environments/<environment>/modules/elasticsearch/lib/puppet/type/elasticsearch_role.rb': Could not autoload puppet/provider/elasticsearch_role/shield: undefined method `shield_config' for Puppet::Type::Elasticsearch_role::ProviderShield:Class
Error: Could not autoload puppet/provider/elasticsearch_role_mapping/shield: undefined method `shield_config' for Puppet::Type::Elasticsearch_role_mapping::ProviderShield:Class
Error: Failed to load custom type 'elasticsearch_role_mapping' from '/etc/puppetlabs/code/environments/<environment>/modules/elasticsearch/lib/puppet/type/elasticsearch_role_mapping.rb': Could not autoload puppet/provider/elasticsearch_role_mapping/shield: undefined method `shield_config' for Puppet::Type::Elasticsearch_role_mapping::ProviderShield:Class
Error: Could not autoload puppet/provider/elasticsearch_user_file/shield: undefined method `shield_config' for Puppet::Type::Elasticsearch_user_file::ProviderShield:Class
Error: Failed to load custom type 'elasticsearch_user_file' from '/etc/puppetlabs/code/environments/<environment>/modules/elasticsearch/lib/puppet/type/elasticsearch_user_file.rb': Could not autoload puppet/provider/elasticsearch_user_file/shield: undefined method `shield_config' for Puppet::Type::Elasticsearch_user_file::ProviderShield:Class
Error: Could not autoload puppet/provider/elasticsearch_user_roles/shield: undefined method `shield_config' for Puppet::Type::Elasticsearch_user_roles::ProviderShield:Class
Error: Failed to load custom type 'elasticsearch_user_roles' from '/etc/puppetlabs/code/environments/<environment>/modules/elasticsearch/lib/puppet/type/elasticsearch_user_roles.rb': Could not autoload puppet/provider/elasticsearch_user_roles/shield: undefined method `shield_config' for Puppet::Type::Elasticsearch_user_roles::ProviderShield:Class

with the command used: /opt/puppetlabs/bin/puppet generate types --environment nxms_production --force

[jameskirsop]

FWIW, I've tried to deploy Elasticsearch 7.x to a clean RHEL 8 server and the service won't start. The README is confusing as it mentions support in some places for 7, but not others.

Some clear details in the README on what is supported, and perhaps some clarification here, from @fatmcgav, would be really helpful

[sferry]

HEllo, I am asking here about my case:
I have a functionning ELK system which I have updated to elk7.14 with the module version 6.4 (just changing JVM options) (4servers for 3 elasticsearch +1 logstash) .
This is confusing:
The module 7.x has breaking changes. But what about the 8.x ? Can I update to 8, and update to elk7.17 ?
Thanks for your help!
Sophie