During transition period logging format should be identical to the one in the current Kibana

[azasypkin]

During current platform -> new platform transition period logging format should be identical to one that used by the current platform.

We'll likely need an additional temporary Layout (s) to accomplish that.

[kimjoar]

We also need to read the old log config instead of the new one. Maybe we can create a "LegacyLoggingConfig" that can read the old config and when valid return a new config. (Or something like that which is specifically built to be easily removed later)

[azasypkin]

It seems there are two possible approaches here:

1. Introduce LegacyPatternLayout and LegacyJsonLayout that will format messages in the same way as it's done in the legacy platform. In this case all log records in the new platform will be managed by new platform file/console appenders that will write to file/console on their own independently from the legacy platform;

2. Introduce LegacyLogAppender that will just forward destructured LogRecord to kbnServer.server.log (haven't checked that yet, but theoretically that should work).

The main benefit of the 1st approach is that we'll be able to mature new logging system and catch issues earlier. But there is a risk that we'll miss something and introduce subtle incompatible changes (e.g. for file based logging we'll have two concurrent writers that's not an issue per se, but still).

The main benefit of the 2nd approach is that the risk and the amount of code needed are minimal.

I'm leaning towards the 1st option, but will definitely understand if the rest of the team thinks that the safer the better :). What do you think @elastic/kibana-platform?

[kimjoar]

Interesting. My immediate thought is that I lean towards the 2nd approach to begin with, mainly because of the risk. Then, when we've given that some time we could maybe switch over to the new logger (and maybe even have all the logs from the old platform sent there? Basically replace the current log setup.)

Would be great to hear @epixa's thoughts on this too.

[epixa]

If our plan is to remove the old log format entirely when we turn off the old code, then I'd be all for option 2, but it doesn't seem wise to me to remove the log formats entirely. Logs are just such a critical part of someone's ops scenario with Kibana. If we do want to get rid of them entirely, I'd propose a process like this:

1. 6.x - introduce alternative log formatters that we prefer
2. 7.0 - change default log formatter to the one we prefer, old one called legacy_json or something
3. 8.0 - remove legacy_json

If we did something like that, then we must implement the legacy behavior in the new platform.

[azasypkin]

If our plan is to remove the old log format entirely when we turn off the old code

One of the goals that new logging subsystem should achieve is unifying of default log formats across the stack assuming that ES log format is our "reference" format. So yeah, eventually default Kibana and ES logs will look the same. Logging format for both ES and Kibana (in new platform only) is configurable, so users will be able to change format if they want to.

it doesn't seem wise to me to remove the log formats entirely.

Agree, and there is no plan to remove the old format entirely any time soon :) With both options we'll have logs in the old format.

The question was more like: who should write logs coming from the new platform to stdout/file right now. With the 1st option new platform will manage its own logs independently from the old one (with custom legacy "layouts" that will ensure that logs are output in the old format). With the 2nd option we just forward logs from new platform to the old one and it does what it does currently (formatting and writing).

1. 6.x - introduce alternative log formatters that we prefer
2. 7.0 - change default log formatter to the one we prefer, old one called legacy_json or something
3. 8.0 - remove legacy_json

Sounds like a good plan, 2nd step will require custom legacy "layouts" I mentioned in the 1st option. Also I hope that before K8.0 we'll allow plugins to register their own logging appenders and layouts, so that users can tune logging according to their needs even after we remove legacy-json and legacy-pattern.

If we did something like that, then we must implement the legacy behavior in the new platform.

Yeah, as I said above, that's exactly the plan.

[epixa]

Yeah, as I said above, that's exactly the plan.

If option 1 was already a done deal, I must be misunderstanding what sort of feedback you were looking for :)

In any case, it sounds like we're all in agreement here.

[azasypkin]

If option 1 was already a done deal, I must be misunderstanding what sort of feedback you were looking for :)

Ah, sorry, I know, I'm bad at asking questions :) I was just asking whether we want to take a risk and do new-platform logging in the new-platform itself (use new-platform logging in full and just imitate old log format) or we want to reduce the risk and leave new-platform log formatting and writing to the old-platform.

Anyway, based on @kjbekkelund's and your replies I feel like we should choose 2nd (safer) option right now and borrow some parts of option 1 later on :)

[azasypkin]

Fixed via 01d0f31