Add support for Socket Logging Handler with basic ECS format logging

[troosan]

This pull request is a followup of PR #23128 and is greatly inspired by the following provided by @geoand https://github.com/quarkusio/quarkus/compare/main...geoand:%2323127?expand=1

It allows sending logs in ECS format to a socket (typically a logstash tcp input)

fixes #23127 and answers my own question on StackOverFlow

[troosan]

I'm still working on updating the documentation for it (logging and centralized-log-management)

The logging one is easy, the centralized log management is more complicated, as it promotes the use of the GELF format.
I think I'll just add a section under "GELF alternative: use Syslog" for now.

But I believe logging in ECS format should be promoted over the GELF format as it fits better in the ELK stack

[geoand]

Looks good, thanks!

@dmlloyd any comments?

[github-actions]

🙈 The PR is closed and the preview is expired.

[troosan]

The resulting json is the following

{
            "threadId": 449,
           "timestamp": "2024-09-12T08:56:17.609151+02:00",
          "@timestamp": 2024-09-12T06:56:17.610853593Z,
          "threadName": "Shutdown thread",
             "message": "my-app stopped in 0.118s",
                 "ndc": "",
            "sequence": 4335,
     "loggerClassName": "org.jboss.logging.Logger",
          "loggerName": "io.quarkus",
         "processName": "/Users/antoine/.sdkman/candidates/java/17.0.9-graalce/bin/java",
           "processId": 65262,
               "level": "INFO",
            "@version": "1",
                "mdc": {},
            "hostName": "macbook-m1-pro.localdomain"
}

I guess it's a good start, but we could add default values for parameters as serviceName, serviceVersion, serviceEnvironment, ...

https://www.elastic.co/guide/en/ecs-logging/java/1.x/setup.html#_add_the_dependency

or we let the generation of an ECS compatible json to the https://github.com/quarkiverse/quarkus-logging-json/blob/main/runtime/src/main/java/io/quarkiverse/loggingjson/LoggingJsonRecorder.java#L92 extension.

I'm not sure why there are 2 impl for json logging actually.

[troosan]

@dmlloyd @geoand should we natively provide an valid ECS json?

• using an external library (provided by elastic) It looks quite lightweight: https://github.com/elastic/ecs-logging-java/tree/main/jboss-logmanager-ecs-formatter
• by building the json manually

or should I update the documentation to mention the quarkiverse version of the quarkus-logging-json, which supports ECS encoding. This will need a PR for that extension though to enable using the socket handler.

[geoand]

@loicmathieu you've used the logging-json stuff IIRC, what's your take?

[loicmathieu]

quarkus-logging-json is the official way to log in JSON.

What I don't understand is why it mandates modification in the logging-json library, socket loggings should be a log handler so it should work with all supported log format/pattern?

By the way, logging-gelf already support that.

[troosan]

I think the reason this required a change in the logging-json is because it's configured in a different way.
The json logging can be configured per handler (console, file and now socket) whereas the gelf handler handles only it's own format.
Maybe indeed this should be reversed, and the socket handler should allow only the ECS format, just as the GELF handler is allowing only the GELF format.

I know the gelf logging already supports sending logs to logstash, but my operations team is not too keen on enabling & maintaining another input (and the filters that go with it) in the logstash pipeline.

[loicmathieu]

OK then.

[troosan]

@geoand @gsmet Contrary to what I put int the documentation, with this current PR, the format of the JSON does not follow the ECS standards.

For now I have gone around this by setting field key overrides through the configuration, hiding/adding additional fields, like this:

quarkus.log.socket.json.exception-output-type=formatted
quarkus.log.socket.json.key-overrides=timestamp=@timestamp,\
	logger_name=log.logger,\
	level=log.level,\
	thread_name=process.thread.name,\
	thread_id=process.thread.id,\
	host_name=host.hostname,\
	stack_trace=error.stack_trace
quarkus.log.socket.json.excluded-keys=sequence,loggerClassName
quarkus.log.socket.json.additional-field."service.environment".value=${quarkus.profile}
quarkus.log.socket.json.additional-field."service.name".value=${quarkus.application.name}
quarkus.log.socket.json.additional-field."service.version".value=${quarkus.application.version}
quarkus.log.socket.json.additional-field."data_stream.type".value=logs
quarkus.log.socket.json.additional-field."ecs.version".value=1.12

This is only however not a very clean solution, I could also do this programmatically like this in the LoggingJsonRecorder

    public RuntimeValue<Optional<Formatter>> initializeSocketJsonLogging(JsonLogConfig config) {
        if (config.socketJson.logFormat == JsonConfig.LogFormat.ECS) {
            addEcsFields(config.socketJson);
        }
        return getFormatter(config.socketJson);
    }

    private void addEcsFields(JsonConfig config) {
        EnumMap<Key, String> keyOverrides = PropertyValues.stringToEnumMap(Key.class, config.keyOverrides.orElse(null));
        keyOverrides.putIfAbsent(Key.TIMESTAMP, "@timestamp");
        keyOverrides.putIfAbsent(Key.LOGGER_CLASS_NAME, "log.logger");
        keyOverrides.putIfAbsent(Key.LOGGER_NAME, "log.logger");
        keyOverrides.putIfAbsent(Key.LEVEL, "log.level");
        keyOverrides.putIfAbsent(Key.PROCESS_ID, "process.pid");
        keyOverrides.putIfAbsent(Key.PROCESS_NAME, "process.name");
        keyOverrides.putIfAbsent(Key.THREAD_NAME, "process.thread.name");
        keyOverrides.putIfAbsent(Key.THREAD_ID, "process.thread.id");
        keyOverrides.putIfAbsent(Key.HOST_NAME, "host.hostname");
        keyOverrides.putIfAbsent(Key.SEQUENCE, "event.sequence");
        keyOverrides.putIfAbsent(Key.EXCEPTION_MESSAGE, "error.message");
        keyOverrides.putIfAbsent(Key.STACK_TRACE, "error.stack_trace");
        config.keyOverrides = Optional.of(PropertyValues.mapToString(keyOverrides));

        config.additionalField.computeIfAbsent("ecs.version", k -> buildFieldConfig("1.12.2", Type.STRING));
        config.additionalField.computeIfAbsent("data_stream.type", k -> buildFieldConfig("logs", Type.STRING));

        quarkusConfig.getOptionalValue("quarkus.application.name", String.class).ifPresent(
                s -> config.additionalField.computeIfAbsent("service.name", k -> buildFieldConfig(s, Type.STRING)));
        quarkusConfig.getOptionalValue("quarkus.application.version", String.class).ifPresent(
                s -> config.additionalField.computeIfAbsent("service.version", k -> buildFieldConfig(s, Type.STRING)));
        quarkusConfig.getOptionalValue("quarkus.profile", String.class).ifPresent(
                s -> config.additionalField.computeIfAbsent("service.environment", k -> buildFieldConfig(s, Type.STRING)));
    }

which is not very clean either.

Also missing is the addition of fields like trace.id and span.id instead (or in addition to) of having them in the mdc.traceId and mdc.spanId

So question is, shall I commit this working (but far from perfect) solution and adapt the documentation accordingly?
Or do I just specify in the documentation how to make it ECS compatible through properties?

[geoand]

Or do I just specify in the documentation how to make it ECS compatible through properties?

I think this is best for now. @loicmathieu WDYT?

[troosan]

@geoand in case we go for the code solution instead of the properties solution, do you know how I can get the traceId and spanId? Those will obviously only be available if the OpenTelemetry dependency is present. Is there a way to retrieve those two values without relying on the io.opentelemetry.context classes?
I would like to add those in the ECS tracing fields
My current solution is to copy them in the Logstash pipeline with the following filter but it would be nicer to natively support it.

filter {
  if ![span][id] and [mdc][spanId] {
    mutate { rename => { "[mdc][spanId]" => "[span][id]" } }
  }
  if ![trace][id] and [mdc][traceId] {
    mutate { rename => {"[mdc][traceId]" => "[trace][id]"} }
  }
}

[geoand]

@geoand in case we go for the code solution instead of the properties solution, do you know how I can get the traceId and spanId? Those will obviously only be available if the OpenTelemetry dependency is present. Is there a way to retrieve those two values without relying on the io.opentelemetry.context classes?

@brunobat can give you the best way to access those, although in order to make reduce unnecessary couplinh, we'd need to put some kind of SPI in place.

[brunobat]

@troosan You need access to the context like the MDCEnabledContextStorage does:

quarkus/extensions/opentelemetry/runtime/src/main/java/io/quarkus/opentelemetry/runtime/MDCEnabledContextStorage.java

Line 30 in bbd821a

OpenTelemetryUtil.setMDCData(beforeAttach, null);

But this data is maintained by us in the MDC context, therefore, your solution sounds ok.

[loicmathieu]

I think we should have a util for that that can extract the spanId/transactionId if present as we need to do it for all handlers. For ex it would be useful to be able to forward these ids to Google Cloud Loggin, today it must be done manually by implementing a trace extractor: https://docs.quarkiverse.io/quarkus-google-cloud-services/main/logging.html#_tracing

This is a cross-extension concern so we should have something in the core that didn't depends on OTEL.

@troosan

Or do I just specify in the documentation how to make it ECS compatible through properties?

Yes, again we may want to provide something that will work cross handler, for example this is the code we have for Google Cloud Logging: https://github.com/quarkiverse/quarkus-google-cloud-services/blob/main/logging/runtime/src/main/java/io/quarkiverse/googlecloudservices/logging/runtime/ecs/EscJsonFormat.java

[troosan]

@loicmathieu that is indeed exactly what we need, Ideally the generation of the json (ECS or not) and where to send it (Google, Logstash, ...) should be decoupled. Otherwise we're just duplicating code all over.

[brunobat]

This is a cross-extension concern so we should have something in the core that didn't depends on OTEL.

But all tracing is currently impl by OTel...
What we can have is some service you can inject and get the Otel tracing data in a simple way.

[troosan]

@geoand I have updated the documentation on how to make the json ECS compatible through properties.
https://quarkus-pr-main-43232-preview.surge.sh/version/main/guides/centralized-log-management#logstash_ecs

Do you think this could get merged as is in next iteration? (I can squash the commits before merging)

Longer term the logging facility probably needs to be reviewed to avoid unnecessary duplication across the extensions.

[gsmet]

I added some minor suggestions and questions.

Other than that, I think the patch looks good and it's a very nice contribution.

[dmlloyd]

Sorry for the late review, but if this is being used specifically to talk to one kind of service, then should we be having a general socket config? I think that users might have a harder time configuring a generic socket handler to talk to some service than if they could have a configuration specific to the kind of service they're talking to.

[troosan]

Theoretically the socket to which you are writing is independent of the format in which you send the data to it.
The current setup will allow you to format the logs however you want, JSON is only one of those.

The Socket handler should allow you to send logs to Splunk too for instance. I've not tried it out though.

[dmlloyd]

Won't this break if we have more than one socket handler configured for some reason?

[troosan]

you mean that we should be able to work with "named" handlers? To allow writing logs to 2 different socket appenders? (Logstash and Splunk for instance)

I must admit I don't know. This is just a copy of LogFileFormatBuildItem for instance

[troosan]

I guess everybody was in Antwerp at Devoxx last week :-)
Any chance to have this part of the next iteration?

[dmlloyd]

This I just don't like. Could we at least have a enum property on the JSON formatter for format presets? Asking every user to provide all of this each time is not OK IMO.

[troosan]

that's indeed what I proposed here: #43232 (comment)
@dmlloyd let me know what you prefer

[troosan]

looks like the move of the logging to @ConfigMapping made a mess out of my PR. I'll update it.
@geoand @dmlloyd @gsmet what do you prefer concerning #43232 (comment) ?

[troosan]

@dmlloyd what shall I do for this PR? I personally would tend to agree with you that it's much easier for the enduser to just specify quarkus.log.socket.json.log-format=ECS and do the key-overrides programmatically.
Once someone comes up with a "clean" way to produce the ECS formatted json, we can just swap the impl. The config key can stay the same, so it would be backward compatible.

[dmlloyd]

@dmlloyd what shall I do for this PR? I personally would tend to agree with you that it's much easier for the enduser to just specify quarkus.log.socket.json.log-format=ECS and do the key-overrides programmatically. Once someone comes up with a "clean" way to produce the ECS formatted json, we can just swap the impl. The config key can stay the same, so it would be backward compatible.

Sure, I like this approach. That would cover my two concerns (configurability, plus being able to clean up things in the future).

[quarkus-bot]

Status for workflow Quarkus Documentation CI

This is the status report for running Quarkus Documentation CI on commit 3d110d1.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

Warning

There are other workflow runs running, you probably need to wait for their status before merging.

[quarkus-bot]

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit 3d110d1.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

[troosan]

@dmlloyd changes done and documentation updated accordingly

[dmlloyd]

Looks OK. I think we'll probably want to revisit some of these internals at some point, but you've really done your best to accommodate and I think the current state is probably acceptable for now. Thanks.

[troosan]

you're welcome. I really need to dive into the quarkus build specificities to understand better how all this works !

[cescoffier]

@loicmathieu @dmlloyd Are we good on this one?

[dmlloyd]

I'm good with it

[cescoffier]

Thanks @dmlloyd !

[loicmathieu]

Good for me.

I'll open a discussion so review all ou centralized log solutions now that we have 3 of them: socket appender, logstash gelf and Open telemetry logs we would want to document the differences and maybe deprecate the old logstash gelf.

I'm also have a look if I can reuse the new ECS formatter in Google Cloud Logging