Support for Elastic Common Schema

[rezvanf]

A few questions before you begin:

Is this an issue related to the Serilog core project or one of the sinks or
community projects.
This issue list is intended for Serilog Elasticsearch Sink issues. If this issue relates to another sink or to the code project,
please log on the related repository. Please use Gitter chat and Stack Overflow for discussions and questons.

Does this issue relate to a new feature or an existing bug?

• Bug
• New Feature

What version of Serilog.Sinks.Elasticsearch is affected? Please list the related NuGet package.
latest nuget

What is the target framework and operating system? See target frameworks & net standard matrix.

• netCore 2.0
• netCore 1.0
• 4.7
• 4.6.x
• 4.5.x

Please describe the current behavior?
n/a

Please describe the expected behavior?
Introduce fluent api to enable ecs compliant fields to be logged

Introduce a json sink that writes out compliant ecs json straight to elastic search.

I have already started an implementation of this in my organisation and would be happy to add this functionality through a PR.

If the current behavior is a bug, please provide the steps to reproduce the issue and if possible a minimal demo of the problem

[mivano]

I had not heard about Elastic Common Schema, but it sounds very useful! Would certainly welcome a PR as long as it does not break older clients and can be switched on or off. Curious to see what you come up with!

[rezvanf]

Great. Will do.

This is ecs btw.

https://www.elastic.co/blog/introducing-the-elastic-common-schema

[mivano]

More reading to do this weekend :-) Nice move to a standard which will allow easier dashboarding and queries, I like it.

[mivano]

@rezvanf any progress on this ECS? Are you blocked somewhere?

[rezvanf]

So I have a 1st draft i am just working with my organisation on how i can contribute back without breaking my organisations policies.

[spfaeffli]

@mivano, @rezvanf any news on this?

[rezvanf]

Yes I am working on to put a PR through. I have a version of the code which supports a json formatter with enrichers, which support some of the ECS fields. The output i have had reviewed directly with Elastic themselves. This has taken me some time because the code is coming through the organisation i work for and therefore I need to consider the IP implications. Can you please help and tell me where the contributing guide is ..I would like to know which branch i need to put my PR to.

[rezvanf]

I have just noticed it is the dev branch that the PR should go into.

The current elastic search sinks targets .net 1.1 is this something we wish to keep?

As 1.1 will reach of end of support in 20 days.

[mivano]

Sorry, yes use dev as a start branch. Then I can create test packages. It also now contains an updated version that removed 1.1 support.

[lancegliser]

Hi @rezvanf, I'm coming up against this problem myself. Have any progress or base pieces we could work from?

[rezvanf]

I am attempting to build my projects that I have added to the existing solution. My projects target 2.2 and I am getting this error
Severity Code Description Project File Line Suppression State

Error NETSDK1045 The current .NET SDK does not support targeting .NET Core 2.2. Either target .NET Core 2.1 or lower, or use a version of the .NET SDK that supports .NET Core 2.2. Serilog.Formatting.Elasticmiddleware C:\Program Files\dotnet\sdk\2.1.505\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.TargetFrameworkInference.targets 137

[rezvanf]

This is the installed SDK
.NET Core SDKs installed:
2.1.403 [C:\Program Files\dotnet\sdk]
2.1.502 [C:\Program Files\dotnet\sdk]
2.1.505 [C:\Program Files\dotnet\sdk]
2.1.602 [C:\Program Files\dotnet\sdk]
2.2.100 [C:\Program Files\dotnet\sdk]

[ledouxpl]

@rezvanf @mivano
Any update on the PR? I would gladly contribute if a feature branch is made public.

[mivano]

Sorry, I do not have any more information. @Mpdreamz are you also still interested in this development?

[Mpdreamz]

Hi! Yes we will be opening up a repository in the coming week that will aid with this effort.

We will take a more active and more importantly open approach following that, anyone's peering eyes and contributions will be very much appreciated.

cc @codebrain

[rezvanf]

i have a working solution, i just have the challenge of getting from my place of work into the outside world here on github. It has to go through IP checks, which is time consuming unfortunately.

[mhoyer]

Just for the sake of completeness, these are the links to the ESC field references:

• https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html
• https://github.com/elastic/ecs/blob/master/schema.json

Maybe we can generate the C# sources from that schema.json?

In addition I found this repo: https://github.com/ehsan-maleki/serilog-sinks-elastic-ecs-logstash

[Mpdreamz]

Hi everyone,

We've opened our repos that we have for ECS to be used with .NET

https://github.com/elastic/ecs-dotnet

It includes a generator that can emit C# sharp types from the ECS schema.

The types can serialize/deserialize by themselves to an ecs json log format. We are still debating what that means exactly but for now it makes sure certain properties are always first (timestamp, log.level, message) so that if persisted to file it remains somewhat scannable.

The repos also adds a Serilog TextFormatter that can be used with many different sinks. We try to play nice with Serilog.Enrichers.* packages OOTB.

We expect to release initial beta's in the coming weeks.

[mivano]

Great progress! The TextFormatter is a nice solution. Thanks for the update.

[orjan]

@Mpdreamz this formatter is really awesome and together with data streams we're having a sensible default almost out for the box! Is there anything more we'll need to do in this sink more that setting the Elastic.CommonSchema.Serilog.EcsTextFormatter to fully support it?

@mivano I didn't know about this formatter, what about promoting it a little bit more in the readme?

[orjan]

I'm not sure if this is related to this sink, the formatter or something lacking in Kibana? But being able to filter on values from serilog is important.

References

• fields starting with underscore ( _ ) doesn't show. elastic/kibana#2551
• [FEATURE] Serilog add custom and default fields elastic/ecs-dotnet#73

[mivano]

Totally in favor of promoting this more, however, my knowledge is pretty limited. But it looks really interested and a nice way forward for the ES sink.