Type_name dynamic

[LilianDurand]

I want to have different type according to a field of my records.

The issue is that elasticsearch seems to create a mapping only for the first record it receives but I want to record all my logs in the same index everyday but in different types.

Here is my code, if someone can help me.
Thank's.

<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  tag filter
  source_hostname_key "ip_host"
  priority_key "priority"
  facility_key "facilityLevel"

  <parse>
    name syslog
    @type regexp
    expression /^(?<syslogTime>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? (?<error>[a-zA-Z0-9_]*)?(?:[^\:]*\:)? *(?<message>.*)$/
    time_key syslogTime
    time_format "%b %d %H:%M:%S"
    time_type string
    keep_time_key "true"
    types syslogTime:time:%b %d %H:%M:%S
    include_tag_key "true"
  </parse>

</source>

<match filter.**>
  @type elasticsearch_dynamic
  hosts xxxx

  <buffer>
    type memory
    flush_interval 10s
    queue_limit_length 1024
    flush_thread_count 16
  </buffer>

  type_name fluentd_${record['error']}
  index_name junos_log_${Time.new(Time.now.year, Time.now.month, Time.now.day).to_i.to_s}
</match>

[cosmo0920]

Please show us your using Elasticsearch, Fluentd, and fluent-plugin-elasticsearch version?

[LilianDurand]

I'm using td-agent version 0.16.1, elasticsearch v 6.4.0 and fluent-plugin-elasticsearch 2.10.3

[cosmo0920]

td-agent 0.16.1 is nonexistent version....
https://support.treasuredata.com/hc/en-us/articles/360001479187-The-td-agent-ChangeLog

[LilianDurand]

Sorry, version 1.2.2

[cosmo0920]

You mean you use Fluentd 1.2.2, right?

[LilianDurand]

I'm pretty sure, I've installed td agent 1 month ago with the latest version I think and when I execute the command line "td-agent --version" I see in return "td-agent 1.2.2"
But if I execute the command line "fluentd --version" I see "fluentd 1.2.5"

[cosmo0920]

td-agent1 series is already EOL.
Could you confirm current running td-agent version?

[LilianDurand]

I'm currently using td-agent version 3.2.0

[cosmo0920]

elasticsearch v 6.4.0 and fluent-plugin-elasticsearch 2.10.3

This is because Elasticsearch v6.0.0 or above limitation.

Indices created in Elasticsearch 6.0.0 or later may only contain a single mapping type. Indices created in 5.x with multiple mapping types will continue to function as before in Elasticsearch 6.x. Mapping types will be completely removed in Elasticsearch 7.0.0.

ref: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html#_index_per_document_type

[cosmo0920]

Root cause is: elastic/elasticsearch#24317