ILM policy creation failed

[plejik]

Problem

Hey try to create index template with ilm policy and stuck on ilm creation phase
Forsome reason i get error like

2023-11-24 14:04:05 +0000 [info]: #0 PUT https://elastic:*********@localhost:9200/_enrich/policy/test-policy [status:400, request:0.528s, query:N/A]
2023-11-24 14:04:05 +0000 [debug]: #0 > {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"3d","max_primary_shard_size":"7gb"},"set_priority":{"priority":100}},"min_age":"0ms"},"delete":{"min_age":"4d","actions":{"delete":{}}}}}}
2023-11-24 14:04:05 +0000 [debug]: #0 < {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:12] [policy] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:12] [policy] unknown field [phases]"},"status":400}
2023-11-24 14:04:05 +0000 [fatal]: #0 [400] {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:12] [policy] unknown field [phases]"}],"type":"x_content_parse_exception","reason":"[1:12] [policy] unknown field [phases]"},"status":400}

Try to figure out why plugin try access _enrich/ endpoint but not _ilm/
Find out only code where it was called like https://github.com/uken/fluent-plugin-elasticsearch/blob/master/lib/fluent/plugin/elasticsearch_index_lifecycle_management.rb#L70

Steps to replicate

Test docker

docker run --rm  -u fluent --name fluentd  --network host  -v fluentd:/fluentd/etc -v /temp:/var/log/containers/ fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch8-1 -c /fluentd/etc/fluent.conf

config

<source>
  @type tail
  path /var/log/containers/nginx*.log
  pos_file /tmp/fluentd-docker.pos
  read_from_head true
  tag kubernetes.*
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_type string
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      keep_time_key false
    </pattern>
    <pattern>
      format regexp
      expression /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
      time_format '%Y-%m-%dT%H:%M:%S.%N%:z'
      keep_time_key false
    </pattern>
  </parse>
</source>

<filter kubernetes.var.log.containers.**.log>
  @type kubernetes_metadata
</filter>

<filter kubernetes.var.log.containers.**>
  @type parser
  key_name log
  <parse>
    @type regexp
    expression /^(?<ng.remote_addr>[^ ]*) - (?<ng.remote_user>[^ ]*) \[(?<ng.time_local>[^\]]*)\] "(?<ng.request_method>\S+)(?: +(?<ng.request_path>[^ ]*) +\S*)?" (?<ng.status>[^ ]*) (?<ng.body_bytes_sent>[^ ]*)(?: "(?<ng.http_referer>[^\"]*)" "(?<ng.http_user_agent>[^\"]*)"(?:\s+(?<ng.http_x_forwarded_for>[^ ]+))?) "(?<ng.upstream_addr>[^ ]*)" "(?<nginx.access.upstream_status>[^ ]*)" "(?<nginx.access.upstream_response_time>[^ ]*)" "(?<nginx.access.request_time>[^ ]*)" "(?<http.request.bytes>[^ ]*)" "(?<nginx.access.origin>[^ ]*)" "(?<nginx.access.x_requested_with>[^ ]*)"$/
    time_format %d/%b/%Y:%H:%M:%S %z
  </parse>
</filter>

# <match **>
#   @type stdout
# </match>

<match kubernetes.**>
  @type elasticsearch
  host "localhost"
  port 9200
  @log_level debug
  user "elastic"
  password "xxxxxxx"
  scheme https
  ssl_verify false
  reload_on_failure true
  logstash_prefix nginx
  with_transporter_log true
  logstash_format true
  time_key time
  enable_ilm true
  ilm_policy_id test-policy
  ilm_policy_overwrite true
  ilm_policy {
    "policy": {
        "phases": {
        "hot": {
            "actions": {
            "rollover": {
                "max_age": "3d",
                "max_primary_shard_size": "7gb"
            },
            "set_priority": {
                "priority": 100
            }
            },
            "min_age": "0ms"
        },
        "delete": {
            "min_age": "4d",
            "actions": {
            "delete": {}
                }
            }
        }
    }
  }
  template_overwrite true
  template_name nginx
  use_legacy_template false
  template_file /fluentd/etc/template.json
  <buffer>
      flush_thread_count 8
      flush_interval 1s
      chunk_limit_size 10M
      queue_limit_length 16
      retry_max_interval 30
      retry_forever true
    </buffer>
</match>

Template json

  "index_patterns": ["nginx-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "priority": 2,
  "_meta": {
    "description": "Nginx Template"
  }
}

Version of plugin 5.3.0 in container

Expected Behavior or What you need to ask

ILM policy created but now i face error above

Using Fluentd and ES plugin versions

• Docker
• fluentd 1.16.3

fluent-gem list | grep es
addressable (2.8.5)
digest (default: 3.1.0)
fluent-plugin-kubernetes_metadata_filter (3.2.0)
mime-types (3.5.1)
mime-types-data (3.2023.1003)
resolv (default: 0.2.1)
resolv-replace (default: 0.1.0)
rest-client (2.1.0)
fluent@riven:~$ fluent-gem list | grep elastic
elastic-transport (8.3.0)
elasticsearch (8.11.0)
elasticsearch-api (8.11.0)
fluent-plugin-elasticsearch (5.3.0)

• 8.11.0
• ES template(s) (optional)

[Killroad]

Dealing with the same problem right now. So far the only working solution for me has worked is to revert back to version fluent-plugin-elasticsearch v5.2.2, which was in ghcr.io/banzaicloud/fluentd:v1.14.6-alpine-5, which was set by default for helm chart logging-operator v3.17.9. I also tried this version https://github.com/kube-logging/fluentd-images/pkgs/container/fluentd/94143520?tag=v1.14.6, but it has the same problems since it already has fluent-plugin-elasticsearch v5.3.0

[plejik]

@Killroad yeah that what i am definitely wanna try next :) will test this image as well in my setup Thx for comment, also i checked code ( as far my knowledge goes ) and looks like in data_ingest_ilm something code looks similar
Update: It's worked just add suppress_type_name true option

[plejik]

@cosmo0920 if possible could you please help here if possible, cause unfortunately i not so familiar with ruby ( i tried do local code change and build gem + fluentd container - but failed, definitely need more time with that :D ). Thx

[Killroad]

@plejik Regarding the suppress_type_name, in my case it was already set to true, but until I reverted to v5.2.2 there was no result. We have discussed this with the team, and for the moment we are ok with the current solution, but I will hope that in the future it will be fixed for the latest version of plugin. By the way, hello from Dnipro city 😃

[cosmo0920]

Thanks for your report. I tried to fix this issue in #1036.