Auto-registered template doesn't support ElasticSearch 5 string types.

[plmaheu]

Does this issue relate to a new feature or an existing bug?

• Bug
• New Feature

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package.
5.4

What is the target framework and operating system? See target frameworks & net standard matrix.

• netCore 2.0
• netCore 1.0
• 4.7
• 4.6.x
• 4.5.x

Please describe the current behavior?
The provided dynamic mapping for strings doesn't work for ElasticSearch 5.x. The raw field isn't being mapped at all.

It's using string type, as in

 "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "fields": {
                  "raw": {
                    "ignore_above": 256,
                    "index": "not_analyzed",
                    "type": "string"
                  }
                },
                "index": "analyzed",
                "omit_norms": true,
                "type": "string"
              }

Please describe the expected behavior?
Should use 'text' and 'keyword', as per 5.x syntax.

[mivano]

Thanks for reporting. Something we need to fix. Care for a PR?

[quetzalcoatl]

I just tried to make it work with ES 6.0, and there's more to it than just the string->text replacement. It seems that 'string' should stay at least in match_mapping_type, I got some errors when I changed it there.

Furthermore, all index are now boolean, so "analyzed/not_analyzed" have to be changed to true/false respectively.

Next thing, one I lack knowledge to properly fix, is _all - it was deprecated in 5.x, and now in 6.x it has been removed. I found information that it is possible to emulate it very well using dynamic templates that match all fields and making them copy_to their contents to some other new field (with custom name and index:true,store:false) that now can be used in lieu of non-existent _all - however I failed trying to configure that.

Last thing I noticed is .. that in 6.x the _default_ has been deprecated. It will work for some time and then evaporate like _all did. The rationale is that now there's a single type per index, so default has no sense. OK. But I'm totally new to ES so I have no idea how to fix that here in the mapping :)

Oh, and IIRC the string->text change will make the mapping incompatible with ES2.x.. so some way of detecting and switching mappings would be needed.. or at least config flag saying use-old-es2-dialect

Anyways, I was able to conjure a custom mapping that writes to ES6.0, much of that was copied from current dev branch:

public static class ElasticSearch
{
    public static ElasticsearchSinkOptions BuildSerilogES60Config(string appname, string eshost)
    {
        return new ElasticsearchSinkOptions(new Uri(eshost))
        {
            AutoRegisterTemplate = true,
            ..... // any other custom settings
            // I've installed newest ElasticSearch 6.0 which does not handle few things used in Serilog.ElasticSearch
            // - in 'properties', 'string' datatype is now called 'text'
            // - 'index' is now true/false, not 'analyzed'/'not_analyzed'
            // - "Enabling [_all] is disabled in 6.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field."
            GetTemplateContent = () => new
            {
                template = ..... , // your IndexName with {0:} replaced with an asterisk "*"
                settings = new Dictionary<string, string>
                {
                    {"index.refresh_interval", "5s"}
                },
                mappings = new
                {
                    _default_ = new
                    {
                        dynamic_templates = new List<Object>
                        {
                            //when you use serilog as an adaptor for third party frameworks
                            //where you have no control over the log message they typically
                            //contain {0} ad infinitum, we force numeric property names to
                            //contain strings by default.
                            {
                                new
                                {
                                    numerics_in_fields = new
                                    {
                                        path_match = @"fields\.[\d+]$",
                                        match_pattern = "regex",
                                        mapping = new
                                        {
                                            type = "text",
                                            index = true,
                                            omit_norms = true
                                        }
                                    }
                                }
                            },
                            {
                                new
                                {
                                    string_fields = new
                                    {
                                        match = "*",
                                        match_mapping_type = "string",
                                        mapping = new
                                        {
                                            type = "text",
                                            index = true,
                                            omit_norms = true,
                                            fields = new
                                            {
                                                raw = new
                                                {
                                                    type = "text",
                                                    index = false,
                                                    omit_norms = true,
                                                    // ignore_above = 256
                                                }
                                            }
                                        }
                                    }
                                }
                            }
                        },
                        properties = new Dictionary<string, object>
                        {
                            {"message", new {type = "text", index = true}},
                            {
                                "exceptions", new
                                {
                                    type = "nested",
                                    properties = new Dictionary<string, object>
                                    {
                                        {"Depth", new {type = "integer"}},
                                        {"RemoteStackIndex", new {type = "integer"}},
                                        {"HResult", new {type = "integer"}},
                                        {"StackTraceString", new {type = "text", index = true}},
                                        {"RemoteStackTraceString", new {type = "text", index = true}},
                                        {
                                            "ExceptionMessage", new
                                            {
                                                type = "object",
                                                properties = new Dictionary<string, object>
                                                {
                                                    {"MemberType", new {type = "integer"}},
                                                }
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            },
        };
    }
}

and later:

            logcfg = logcfg.WriteTo.Elasticsearch(
                ElasticSearch.BuildSerilogES60Config(
                    "foo-the-bar",
                    "http://......:9200"
                ));

        Log.Logger = logcfg.CreateLogger();

It's pretty much totally untested, I just noticed it stopped receiving errors from ES and some events were recorded in ES. I wouldn't consider it as PR-ready at all, but maybe it will help someone to get their logging work in some shape on ES6.. or encourage somebody to extend it with some form of "_all" and make a PR :)

[mivano]

Thanks for the pointers. We need to see how we want to support this. If this is breaking between the different ES versions, we might need to detect or offer a way to use the correct one.

[CumpsD]

A PR for this: #142

[mivano]

PR is merged