Serilog.Sinks.Elasticsearch and ElasticSearch 6.1

[pk241011]

Does Serilog.Sinks.Elasticsearch supports ElasticSearch 6.1 yet? Any pre-release?

Ever since we upgraded the cluster to 6.1 we are seeing that the Serilog is not able to create indices on the cluster.

  <appSettings>
    <add key="serilog:using:Elasticsearch" value="Serilog.Sinks.Elasticsearch" />
    <add key="serilog:minimum-level" value="Verbose" />
    <add key="serilog:ApplicationUI.minimum-level" value="Information" />
    <add key="serilog:write-to:Elasticsearch.nodeUris" value="http://starkindustries:9244" />
    <add key="serilog:write-to:Elasticsearch.indexFormat" value="log-ironsuit-{0:yyyy.MM}" />
    <add key="serilog:write-to:Elasticsearch.templateName" value="ironsuit-template" />
    <add key="serilog:write-to:Elasticsearch.autoRegisterTemplate" value="true" />
    <add key="serilog:write-to:Elasticsearch.bufferBaseFilename" value="C:\Data\Temp\ironsuit-logs" />
    <add key="serilog:write-to:Elasticsearch.bufferLogShippingInterval" value="5000" />
    <add key="serilog:using:IdentifierFile" value="Serilog.Sinks.IdentifierFile" />
    <add key="serilog:write-to:IdentifierFile.path" value="C:\Data\Logs" />
    <add key="serilog:logging-error-directory" value="C:\Data\Logs" />
  </appSettings>

Here are the versions of file in use.

 <packages>
  <package id="Elasticsearch.Net" version="5.2.0" targetFramework="net452" requireReinstallation="true" />
  <package id="Serilog" version="2.4.0" targetFramework="net452" requireReinstallation="true" />
  <package id="Serilog.Settings.AppSettings" version="2.1.2" targetFramework="net461" />
  <package id="Serilog.Sinks.Elasticsearch" version="5.0.0" targetFramework="net452" />
  <package id="Serilog.Sinks.File" version="3.2.0" targetFramework="net452" />
  <package id="Serilog.Sinks.IdentifierFile" version="0.2.0" targetFramework="net452" />
  <package id="Serilog.Sinks.Observable" version="2.0.1" targetFramework="net452" />
  <package id="Serilog.Sinks.PeriodicBatching" version="2.1.0" targetFramework="net452" />
  <package id="Serilog.Sinks.RollingFile" version="3.0.1" targetFramework="net452" />
</packages>`

[jnus]

Same problem here. Seems to be a problem with the index template.

When trying to auto register the template, we get the following error:

No handler for type [string] declared on field [message]

From this post, it seems that string has been replace with the type text. When replacing all string types with text, I get the following error:

Failed to parse mapping [default]: Could not convert [message.index] to boolean
Failed to parse value [analyzed] as only [true] or [false] are allowed

So it seems that the value analyzed and not_analyzed is not a valid boolean. Seems fair enough. Replacing all "index": "analyzed" with the value true, emits an error that the _all field is not supported anymore. Not using that one anyway, so deleting that section finally lets me create the template, with some warning though.

#! Deprecation: Deprecated field [template] used, replaced by [index_patterns]
#! Deprecation: [default] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
{
"acknowledged": true
}

The full template I managed to create:

{
    "template": "logstash-*",
    "settings": {
        "index.refresh_interval": "5s"
    },
    "mappings": {
        "_default_": {
            "dynamic_templates": [
                {
                    "numerics_in_fields": {
                        "path_match": "fields\\.[\\d+]$",
                        "match_pattern": "regex",
                        "mapping": {
                            "type": "text",
                            "index": "true",
                            "omit_norms": true
                        }
                    }
                },
                {
                    "string_fields": {
                        "match": "*",
                        "match_mapping_type": "string",
                        "mapping": {
                            "type": "text",
                            "index": "true",
                            "omit_norms": true,
                            "fields": {
                                "raw": {
                                    "type": "text",
                                    "index": "false",
                                    "ignore_above": 256
                                }
                            }
                        }
                    }
                }
            ],
            "properties": {
                "message": {
                    "type": "text", 
                    "index": "true"
                },
                "exceptions": {
                    "type": "nested",
                    "properties": {
                        "Depth": {
                            "type": "integer"
                        },
                        "RemoteStackIndex": {
                            "type": "integer"
                        },
                        "HResult": {
                            "type": "integer"
                        },
                        "StackTraceString": {
                            "type": "text",
                            "index": "true"
                        },
                        "RemoteStackTraceString": {
                            "type": "text",
                            "index": "true"
                        },
                        "ExceptionMessage": {
                            "type": "object",
                            "properties": {
                                "MemberType": {
                                    "type": "integer"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

[mivano]

There are indeed some breaking changes. See also this issue #129

[CumpsD]

This PR fixes the string deprecation: #142

[MJLHThomassen-Eurocom]

+1

[mivano]

PR merged

[pk241011]

I am able to create new indices now with package "Serilog.Sinks.Elasticsearch 5.7.0". Thanks for that.

However I see that if I want to send data into the indices which are already existing it is failing. These indices were created before the Elasticsearch cluster was upgraded to 6.1 from 5.2.x.

As per the Elasticsearch, Elasticsearch 6.x can use indices created in Elasticsearch 5.x.

[mivano]

An existing index has an already applied template and cannot be changed. You will need to create a new index with the new template and move data over if you want to apply the new mappings.

[pk241011]

@mivano I did a small experiment.
This is how it is in our App.config.

<add key="serilog:write-to:Elasticsearch.indexFormat" value="log-HereThere-{0:yyyy.MM}" />

Index log-HereThere-2017.11 exists.

Now if I replace it with

<add key="serilog:write-to:Elasticsearch.indexFormat" value="log-HereThere-2018.02" />

It does not create a new index.

packages.config as of now.

<packages>
  <package id="Elasticsearch.Net" version="6.0.0" targetFramework="net461" />
  <package id="Serilog" version="2.6.0" targetFramework="net461" />
  <package id="Serilog.Settings.AppSettings" version="2.1.0" targetFramework="net452" />
  <package id="Serilog.Sinks.Elasticsearch" version="6.1.0" targetFramework="net461" />
  <package id="Serilog.Sinks.File" version="4.0.0" targetFramework="net461" />
  <package id="Serilog.Sinks.IdentifierFile" version="0.2.0" targetFramework="net452" />
  <package id="Serilog.Sinks.Observable" version="2.0.1" targetFramework="net452" />
  <package id="Serilog.Sinks.PeriodicBatching" version="2.1.1" targetFramework="net461" />
  <package id="Serilog.Sinks.RollingFile" version="3.3.0" targetFramework="net461" />
</packages>

[mivano]

Can you enable the selflog of Serilog and see if it raises something useful?

[repayjason]

This appears to be happening with all of the latest versions. Please see error below (from the SelfLog):

No handler for type [string] declared on field [message]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [string] declared on field [message]"}}

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Elasticsearch.Net" version="6.1.0" targetFramework="net47" />
  <package id="Serilog" version="2.6.0" targetFramework="net47" />
  <package id="Serilog.Enrichers.Environment" version="2.1.2" targetFramework="net452" />
  <package id="Serilog.Enrichers.Process" version="2.0.1" targetFramework="net452" />
  <package id="Serilog.Sinks.Async" version="1.2.0" targetFramework="net47" />
  <package id="Serilog.Sinks.Console" version="3.1.1" targetFramework="net47" />
  <package id="Serilog.Sinks.Elasticsearch" version="6.5.0" targetFramework="net47" />
  <package id="Serilog.Sinks.File" version="4.0.0" targetFramework="net47" />
  <package id="Serilog.Sinks.Literate" version="3.0.0" targetFramework="net47" />
  <package id="Serilog.Sinks.PeriodicBatching" version="2.1.1" targetFramework="net47" />
  <package id="Serilog.Sinks.RollingFile" version="3.3.0" targetFramework="net452" />
</packages>

Elasticsearch version is 6.2.4.

No index exists on the current ES cluster, and it cannot seem to create one.

Below is how we're instantiating the sink:

loggerConfig.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(elasticSearchUrl))
{
    AutoRegisterTemplate = true,
    AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv6,
    IndexFormat = indexFormat,
    EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog
});

[roelandvh]

We appeared to have the same issue: I saw org.elasticsearch.index.mapper.MapperParsingException: No handler for type [string] declared on field [message] in the elasticsearch logfile.

Using Elasticsearch version 6.2.3, Serilog version 2.6.0 and Serilog.Sinks.Elasticsearch version 6.5.0.

Fixed it by specifying "ESv6" for "autoRegisterTemplateVersion" in the configuration.

[mivano]

Old issue, cleaning up