Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugin fails to install default template when using legacy API with Elasticsearch 8.x (or higher) versions. #1108

Closed
mashhurs opened this issue Jan 9, 2023 · 0 comments · Fixed by #1102
Assignees
Labels

Comments

@mashhurs
Copy link
Contributor

mashhurs commented Jan 9, 2023

Issue description

When we upgrade Elasticsearch to 8.x and use legacy template (template_api=>legacy), the plugin resolves the composable style template as a default template and send to legacy API, obviously it doesn't work (see the produced error below).
The discussion with team brought to the idea of using legacy template api with custom template only. However, if ILM (ilm_* settings) is enabled, the plugin crashes. When plugin tries to set ILM settings to provided custom template, the plugin aligns on Elasticsearch version (pseudo: template_settings = version < 8 ? template['settings']: template['template']['settings']). Obviously, with custom flexible template it is hard to guess where ILM settings located.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

  1. set template_api to legacy
  2. connect to any (including cloud) Elasticsearch where its version is 8.x+
  3. run Logstash and check the log

Provide logs (if relevant):

[ERROR] 2022-12-13 23:42:54.690 [Ruby-0-Thread-12: :1] elasticsearch - Failed to install template 
{:message=>"Got response code '400' contacting Elasticsearch at URL 
'https://my-es-host.com:9200/_template/logstashb'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, 
:backtrace=>["/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", 
"/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:412:in `template_put'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:41:in `install'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install_template'", "/grid01/apps/PIPELINES/logstash-
8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch.rb:583:in 
`install_template'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/outputs/elasticsearch.rb:349:in `finish_register'", "/grid01/apps/PIPELINES/logstash-
8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/outputs/elasticsearch.rb:305:in 
`block in register'", "/grid01/apps/PIPELINES/logstash-8.2.3/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.12.1-
java/lib/logstash/plugin_mixins/elasticsearch/common.rb:167:in `block in after_successful_connection'"]}

Acceptance Criteria

  • Legacy template API is used with only custom templates
  • Plugin should not crash, instead provide a clear message/guide what to do next
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants