[core] Log about individual status changes that occur after startup

[spalger]

We just spent a ton of time tracking down a plugin which was updating it's status from degraded to available very slowly. The vast majority of the time was spent just trying to understand how statuses were changing and would have been pretty trivial if we had seen a log line like:

Status of plugin "alerting" is now available (was degraded for 5 minutes)

I think it would be very useful to log these types of messages at an info level when they occur after some initial delay (to avoid a massive flood of messages about status changes as things startup).

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

I agree that having status changes of individual services/plugins logged could make sense, and would probably help for support too.

I'm still a little concerned about the potential verbosity of the thing. As most plugins don't register custom status handlers, their status is computed depending on the statuses of their dependencies.

In practice, that can lead to a lot of consecutive status changes. For example, if the home (data is another example) plugins changes from available to degraded, this will cause most of the plugins to get to the same state, as almost every plugin's dependency tree has either of those somewhere at the bottom of the tree, causing ~= 100 lines outputted.

We could potentially try to group these status changes by trying to identify the 'root' plugin/service that caused the change in the tree (something like 34 plugins changed from available to degraded due to a change of status coming from 'alerting'), but to be honest, this may not be easy, given the observable hell the status plugin currently is.

So, are we fine just outputting a line in the log for each individual status change, even if in most scenario, we'll have dozen of such lines in every status change batch, or do we want more than that?

[spalger]

I think individual log lines for any delayed status update (one that doesn't happening "immediately" after creation or something) sounds reasonable. They're all useful information that we should be making people aware of I think.