[alerting event log] add event log for alert execution and alerts scheduling actions

[pmuellr]

As the next parts of alerting to instrument with event log entries, it seems like alerting activities will be the most interesting thing to do. At a minimum this would be to add an event log entry when an alert executor indicates that actions should be run via scheduleActions(). We probably also want one when the alert execution function is run.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[peterschretlen]

To support the alert details view, and doing things like annotating charts with alert occurrences, we might also want to log

• when a new alert instance is created
• when an alert instance clears

Or status changes in between ( related to #51099 )

[mikecote]

There's also some questions about if we should be tracking execution failures:

• alert execution
• alert action execution

[pmuellr]

There's also some questions about if we should be tracking execution failures:

• alert execution
• alert action execution

For alert execution, it appears we don't catch errors, which we'll need to start doing to generate event log entries with the error. Code here:

kibana/x-pack/plugins/alerting/server/task_runner/task_runner.ts

Lines 168 to 184 in 8d539aa

	const updatedAlertTypeState = await this.alertType.executor({
	alertId,
	services: {
	...services,
	alertInstanceFactory: createAlertInstanceFactory(alertInstances),
	},
	params,
	state: alertTypeState,
	startedAt: this.taskInstance.startedAt!,
	previousStartedAt: previousStartedAt ? new Date(previousStartedAt) : null,
	spaceId,
	namespace,
	name,
	tags,
	createdBy,
	updatedBy,
	});

For alert action execution, the alert code just queues up the actions with task manager, so the only "error" would be in queuing up the work, I think.

There's a missing link, for action execution itself, in that it doesn't have an alert id - that would be nice, and we may need it - but we will be generating an event that includes both the alert and action id when the action is queued to be run. We could figure out how to pass a "producer" or such to action execution to track things like action ids.