Adjust ElasticsearchSinkOptions.NumberOfShards in SeriLog not working in .Net core

[nguyenmy]

Does this issue relate to a new feature or an existing bug?

• Bug
• New Feature

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package.

What is the target framework and operating system? See target frameworks & net standard matrix.

• netCore 2.2

I have a problem when setting NumberOfShards for ElasticSearch while writing log by SeriLog. I do config for Serilog like this in .Net Core

.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(config.ElasticConnectionUrl))
{
AutoRegisterTemplate = true,
IndexFormat = config.ElasticIndex + "-{0:yyyy.MM.dd}",
NumberOfShards = 2,
NumberOfReplicas = 0
}));
But when I query the setting of the created Index in Kibana, the numberOfShards still 5 (default value). Even for NumberOfReplicas won't affect.

I am using ELK stack to trace logs.

Is anyone know why?

[mivano]

You sure the template is applied/updated? As in, when you query it in ES, you do see the correct value? And you cannot change an existing index, so the change will be in effect when you create a new index (like a new day).

[nguyenmy]

@mivano Yes. I know this change will affect for the new index only. But I don't know why it didn't work. If I created an index template, it works well. But I want to do it in SeriLog.
I tried with an index template like this:
PUT _template/abc { "index_patterns": ["abc*"], "settings": { "number_of_shards": "2", "number_of_replicas" : "0" } }
When I checked the new index, it affected.

[mivano]

When you fetch the generated template, does it include the correct number? It is a nullable settings, but I would assume that it outputs zero: https://github.com/serilog/serilog-sinks-elasticsearch/blob/463c878a805153dc3f97f938c2aa46de0bc815e0/src/Serilog.Sinks.Elasticsearch/Sinks/ElasticSearch/ElasticsearchSinkState.cs#L233-L234

[mivano]

Hmm, looks like more people reporting the same issue: #239

[nguyenmy]

@mivano , the generated template shows the default value (shards=5, replicas=2) even though I set them to (shards=2, replicas=0).

[mivano]

I created some additional unit tests (see #303) but they show a correctly created template. So I m unsure what is causing the issue. You have any clues?

[mivano]

Can you add the option RegisterTemplateFailure = RegisterTemplateRecovery.FailSink to your code and try again? Maybe ES throws an exception and it cannot register the template?

[mivano]

I tried a couple of things and I think you will get an exception which you do not see. So try that FailSink option. Creating a template should not break your application, so that it why it is not throwing by default.
Next to that; make sure to specify the correct version of ES you are targetting. And when there is already a template, by default it will not overwrite.

I tried with this combination and that works fine:

  OverwriteTemplate = true, // You normally do not need this unless you want to overwrite the template on each startup
  DetectElasticsearchVersion = true, // Performs a call to detect ES 6 or ES 7
  AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7 // Use ES 7

[ffMathy]

The only of those that is set differently for me is the DetectElasticsearchVersion property, which was set to false for me.

But shouldn't AutoRegisterTemplateVersion (when set explicitly to V7) override the other property? That's how I understood it at least.

I'll try for a couple of days with this setting, and report back.

[mivano]

I think the DetectElasticsearchVersion can get some more love to make it more foolproof, for now you need to explicitly set it to a certain version and the detection can help.

[CaeirOps]

We figure out the same issue here, resolved by adding the follow parameters:

NumberOfShards = 1
OverwriteTemplate = true
AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7