[FEATURE] Serilog add custom and default fields

[LucaWintergerst]

Is your feature request related to a problem? Please describe.
Adding a field when using Serilog is difficult.
From @Mpdreamz You can extend the types to inject new properties but I don't think we expose this in a nice way through serilog yet

Describe the solution you'd like
An easy way to add:

• custom fields
• default fields

Describe alternatives you've considered

Additional context
Sometimes I want to add an extra field to a log that should nest under the existing ECS top level fields. For example, if I want to add event.dataset

However, in same cases I would like to configure this on the Logger so it always logs it by default.

[Yukon]

A concept I have been thinking about is utilizing the different value types to determine how they should be processed.

These are in the context of the root LogEvent._properties:
ScalarValue Mapped to the labels ECS property.
StructureValue Mapped directly into ECS Base.
SequenceValue Mapped into metadata ECS property (unchanged).
DictionaryValue Mapped into metadata ECS property (unchanged).

The advantage of this method is scalar values are then included in event in a more usable fashion compared to the _metadata mapping. Keeping to the purpose of the label field.

It also becomes trivial to add additional properties like event.dataset by creating an object structure that matches the intended output structure. The object can then be logged with an individual event using .ForContext(“event”, { dataset = “sample” }}). Or in the case globally in the logger configuration .Enrich.WithProperty(“event”, new { dataset = “sample” }}).

To enforce strict typing there is no reason the ECS objects could not be used as well instead of an anonymous type. That also brings a technical advantage that a custom formatter could be created (The base object LogEventPropertyValue implements IFormattable) instead of arbitrary mapping. The issue is this would then not support defining custom fields. A hybrid approach could be taken though to support both use cases.

I have some other thoughts on how to handle deconstructed objects but I’m still fleshing out the idea. It could have some impact on how objects are mapped.

[redxeagle]

On top of this feature-wish comes also that there is currently no way to add structured data to any log events with this formatter.

It looks like logstash is using _metadata as "key", "keyword" pairs. which means I can not inject as value a structured object.

[Update] after some configuration in logstash or elastic search we are able to have a structured object inside _metadata. But opened another issue to rename this field anyway. Or go into a discussion of why this field is named like it is. Underscore fields are uncommon. :-)

[rezvanf]

Is this the correct way someone would add in say the container field which doesn't get autopopulated in the same manner as some of the other ECS fields

...

                  var configuration = new EcsTextFormatterConfiguration();
                   configuration.MapCustom(func);

                   config.WriteTo.Console(new EcsTextFormatter(configuration));
               });

  private static global::Elastic.CommonSchema.Base func(global::Elastic.CommonSchema.Base arg1, LogEvent arg2)
     {
         arg1.Container = new global::Elastic.CommonSchema.Container()
         {
             Runtime = "test"
         };

         return arg1;
     }

This gives the output of

{
	"@timestamp": "2021-07-15T10:07:22.8854057+01:00",
	"log.level": "Information",
	"message": "This is a test",
	"metadata": {
		"message_template": "This is a test"
	},
	**"container": {
		"runtime": "test"
	},**
	"ecs": {
		"version": "1.6.0"
	},
	"event": {
		"created": "2021-07-15T10:07:22.8854057+01:00",
		"severity": 2,
		"timezone": "GMT Standard Time"
	},
	"host": {
		"name": "SomeHostName"
	},
	"log": {
		"logger": "Elastic.CommonSchema.Serilog",
		"original": null
	},
	"process": {
		"thread": {
			"id": 4
		},
		"executable": "iisexpress",
		"name": "iisexpress",
		"pid": 43832
	}
}

[odin568]

Is this the correct way someone would add in say the container field which doesn't get autopopulated in the same manner as some of the other ECS fields

...

                  var configuration = new EcsTextFormatterConfiguration();
                   configuration.MapCustom(func);

                   config.WriteTo.Console(new EcsTextFormatter(configuration));
               });

  private static global::Elastic.CommonSchema.Base func(global::Elastic.CommonSchema.Base arg1, LogEvent arg2)
     {
         arg1.Container = new global::Elastic.CommonSchema.Container()
         {
             Runtime = "test"
         };

         return arg1;
     }

This gives the output of

{
	"@timestamp": "2021-07-15T10:07:22.8854057+01:00",
	"log.level": "Information",
	"message": "This is a test",
	"metadata": {
		"message_template": "This is a test"
	},
	**"container": {
		"runtime": "test"
	},**
	"ecs": {
		"version": "1.6.0"
	},
	"event": {
		"created": "2021-07-15T10:07:22.8854057+01:00",
		"severity": 2,
		"timezone": "GMT Standard Time"
	},
	"host": {
		"name": "SomeHostName"
	},
	"log": {
		"logger": "Elastic.CommonSchema.Serilog",
		"original": null
	},
	"process": {
		"thread": {
			"id": 4
		},
		"executable": "iisexpress",
		"name": "iisexpress",
		"pid": 43832
	}
}

With this I was able to populate the fields "service.name", "service.version", "service.node.name" which are in the core ecs structure. What I would like to set is the "service.environment" field which is extended property and does not exist in object model of this dotnet-ecsschema. Any chance I could work around this?

[Mpdreamz]

I think with both MapCustom and #217 you should now be able to introduce additional properties to serilog in various ways.

In addition extending existing field sets should be possible see this comment: #191 (comment)

However as we are gearing up to release 8.4.0 soon the need to so should be very minimal in the future too.

Thanks everyone for raising this and posting workarounds!