Standardised log level numeric values

[codebrain]

The ECS schema describes the log.level field as:

Original log level of the log event.If the source of the event provides a log level or textual severity, this is the one that goes in log.level. If your source doesn't specify one, you may put your event transport's severity here (e.g. Syslog severity).Some examples are warn, err, i, informational.

This is a string property though, so we'd miss out on the numeric capability in ES even if we decided to index the enum number value in .NET.

I think there is a good argument for wanting the numeric equivalent as well, specifically for queries like "show me everything greater than warning". Of course you could always run a "show me everything where log level is either error or critical" but it implies knowledge of the subsequent descriptive names (a favourite interview question of mine was to describe all of the log levels and it was surprising how many developers gave incomplete answers).

I wonder if we should consider either a new property, say log.level_numeric and index that as an integer - allowing the developer to determine the numeric values (i.e. using the values as defined in their ecosystem). Or, for us to define the numeric values and ask developers to map their values to our predefined list?

At least that way we can show a standardised view across all ecosystems and I imagine we can start feeding this into the Kibana UI (for APM too).

The .NET ecosystem uses the following numeric values.

Trace = 0
Logs that contain the most detailed messages. These messages may contain sensitive application data. These messages are disabled by default and should never be enabled in a production environment.

Debug = 1
Logs that are used for interactive investigation during development. These logs should primarily contain information useful for debugging and have no long-term value.

Information = 2
Logs that track the general flow of the application. These logs should have long-term value.

Warning = 3
Logs that highlight an abnormal or unexpected event in the application flow, but do not otherwise cause the application execution to stop.

Error = 4
Logs that highlight when the current flow of execution is stopped due to a failure. These should indicate a failure in the current activity, not an application-wide failure.

Critical = 5
Logs that describe an unrecoverable application or system crash, or a catastrophic failure that requires immediate attention.

I'm struggling to think if there are an other levels (other than say a "non-level" - which you could argue is either 0 or a null value).

[webmat]

log.level already has sister field event.severity for its numeric representation (and therefore sorting). Although your issue just made me realize there was no mention of that in either field's description 🤦‍♂ We'll need to fix that.

The fields are both currently designed for easy capture of whatever source's naming/numbering scheme, without forcing users to map to a different scheme. So we're not going to introduce limitations on these two fields. However if normalization is desired enough, we could add a second field to allow for that. The fact that these are per source means that it the values you describe are standards in the .Net world, you can immediately start using this numbering scheme in event.severity, without waiting on anyone. Nice, isn't it? ;-) Note that this is true for emitting events, but the library should be able to read these fields if they contain other values, when populated by other sources.

We've discussed normalization in the past, and listed known labels from many sources (15-ish values iirc). The normalization we discussed then was not about mapping to different values, but simply to normalize the equivalent values (info, Informational, informational) to the same string (=> informational). This was issue #129. We haven't done anything with that yet, though.

[ebeahan]

The discussion looks to have been concluded.