[UsageCollection] Expose events API

[afharo]

Updated description

I'd be great to expose on the server side an API for plugins to be able to report events in their plugins (i.e.: new ML job created, new alert triggered, etc.).

UsageCollection will store these events internally and report them via telemetry in the best-aggregated way we can come up with (most likely in a breakdown of 7/30/90/total days accums).

---

Original description

In Pulse, we need to expose the capability for other plugins to push data to the existing channels.

We can achieve this by adding APIs to the UsageCollection plugin (or creating a new Plugin) and adding a deprecated flag to the current makeUsageCollector method to discourage the usage of it in the future

[elasticmachine]

Pinging @elastic/pulse (Team:Pulse)

[afharo]

This can be started after #63953 is done

[Bamieh]

How about moving usageCollector to core and use that as the main interface for sending usage with pulse?

usage collection plugin cannot be disabled and makes sense to have it in core.
This would also help us solve these issues #34940 and #32519

We can update the usage collection api to be the interface for registering usage for different channels.

[afharo]

Yeah! I think it might be worth it!
Also related to #56762
/cc @elastic/kibana-platform any thoughts about not moving the usageCollector to core?

[TinaHeiligers]

@afharo @Bamieh Any changes to core require an RFC with the proposed changes. While the original Pulse RFC was accepted, there wasn't much involvement from other teams and it covered a huge scope. We could potentially go ahead with this work as a POC for a new, more focused RFC.

[afharo]

I agree! We need a proper re-thought about internal channels if they make sense. Although that would mean in the remote service maintaining more indices (as far as I understand, that means slight architectural changes in the telemetry cluster). Not sure if that's the way to go.

One thing we can do for sure is to split @Bamieh's comment into another issue: Moving the plugin to core is a different thing vs. adding new APIs any allegedly new telemetry structure.

[TinaHeiligers]

@elastic/kibana-telemetry should we modify this issue to suite our needs for usage collection or close it?

[afharo]

I'm ok with either option. I guess this one might provide more context for the already linked issues.
@TinaHeiligers Do you mean reusing this issue for the "moving to core" or the "report event-based telemetry" bit 😅

[TinaHeiligers]

@afharo, "moving-to-core" by default would make usageCollection available to other plugins 😄

[afharo]

Oh! Hahaha! I see the confusion here: I guess the title should have been:
"[Pulse] UsageCollection: Add new APIs to report to channels".

UsageCollection APIs are already available to any plugin that requires it. Moving it to core will simply remove the need to require it in the kibana.json file, but the APIs would still be the same, so plugins won't be able to proactively send "events" to "channels". I sincerely think we should provide other devs with the ability to report "telemetry events". Then, the telemetry plugin will take care of aggregating them to 7/30/90 days format and will report them.

Since we found this issue already confusing enough that we both got a different understanding out of it, I'd vote for closing it and creating new ones that may seem more straight-forward :)

[afharo]

@TinaHeiligers I updated the description in this issue so maybe we can repurpose it and add it to the meta issue #69291?

[afharo]

Closing this issue as it feels like a duplicate of #81645

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)