Skip to content
This repository has been archived by the owner on Jun 1, 2024. It is now read-only.

Support for data streams Elastic 7.9 #355

Closed
2 tasks done
orjan opened this issue Sep 11, 2020 · 3 comments
Closed
2 tasks done

Support for data streams Elastic 7.9 #355

orjan opened this issue Sep 11, 2020 · 3 comments

Comments

@orjan
Copy link
Contributor

orjan commented Sep 11, 2020

Does this issue relate to a new feature or an existing bug?

  • New Feature

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package.
<PackageReference Include="serilog.sinks.elasticsearch" Version="8.2.0" />

What is the target framework and operating system? See target frameworks & net standard matrix.

  • netCore 3.1.402

Please describe the current behavior?
Data streams is a reliable way to minimize the number of shards in a cluster. Previously you had to setup an ILM, template and a write alias, data streams is more or less the same but endorsed by Elastic an transparent in Kibana and easier to configure.

There a constraint within data stream and it will only support the op_type=create
https://www.elastic.co/guide/en/elasticsearch/reference/master/use-a-data-stream.html#data-streams-bulk-indexing-requests

When trying to index documents from serilog the request looks like this

POST _bulk
{"index":{"_index":"my-data-stream","_type":"_doc"}}
{"@timestamp":"2020-09-11T16:14:17.6858199+02:00","level":"Error","messageTemplate":"Hello data stream from Serilog","message":"Hello data stream from Serilog"}
#! Deprecation: [types removal] Specifying types in bulk requests is deprecated.
{
  "took" : 0,
  "errors" : true,
  "items" : [
    {
      "index" : {
        "_index" : "my-data-stream",
        "_type" : "_doc",
        "_id" : null,
        "status" : 400,
        "error" : {
          "type" : "illegal_argument_exception",
          "reason" : "only write ops with an op_type of create are allowed in data streams"
        }
      }
    }
  ]
}

Please describe the expected behavior?
In order to support data streams we'll need to be able to alter the op_type from index to create then it should work:
https://github.com/serilog/serilog-sinks-elasticsearch/blob/master/src/Serilog.Sinks.Elasticsearch/Sinks/ElasticSearch/ElasticSearchSink.cs#L156-L160

There's also a deprecation warning for the _type removal but that's covered by a separate issue: #345

From an API point of view I can imagine specifying the op_type similar to this?

                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
                {
                    IndexFormat = "my-data-stream",
                    AutoRegisterTemplate = false,
                    BatchOpType = OpType.Create, 

@mivano
Copy link
Contributor

mivano commented Sep 19, 2020

Merged to dev. Closing this issue for now

@mivano mivano closed this as completed Sep 19, 2020
@glacasa
Copy link

glacasa commented Feb 3, 2022

Just adding a note here, because I found this issue while trying to understand how to write in a DataStream.

If somebody is searching how to set the "BatchOpType", the property is called BatchAction

                  .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(uri)
                  {
                      IndexFormat = "my-data-stream",
                      BatchAction= ElasticOpType.Create
                  });

@gustavodaquino
Copy link

For who that wants to run the code above in a ElasticSearch >= 8.0.0, remember to temporarily TypeName=null as referenced here.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants