[Discover] Context aware extension framework development

[tonyghiani]

📓 Summary

To prevent cyclic dependencies between plugins/packages, it is necessary to have still a mechanism to register external dependencies functionalities. Until now, the extension points have been created to customize some specific UI parts of Discover, such as the flyout content, the sidebar, the top nav, etc.

I believe we should switch to a more Discover-opinionated extension system, where high-level features can be optionally registered and strictly when is not possible to push them down into the Discover codebase, but the responsibility to render them somewhere on the UI is delegated to Discover only.
For example, something like this could be an idea of what it could look like:

// In the obs context
discoverRegistry.registerFeature({
  id: 'ai-assistant', // Constrained available features, no generic `flyout` or `header` pointing to UI areas
  render: (discoverDeps) => <ObsAIAssistant />
})

// In the security context
discoverRegistry.registerFeature({
  id: 'ai-assistant',
  render: (discoverDeps) => <SecurityAIAssistant />
})

// Need to inject custom uiSettings service
discoverRegistry.registerFeature({
  id: 'services',
  getServices: (discoverDeps) => ({uiSettings: customUiSettings})
})

Dependency injection for Discover extensions

Regarding how the dependency injection is managed, the extensions registry currently lives inside the discover plugin, which makes the dependency tree look like:

flowchart TD

B(Logs Explorer) --> A[Discover]
C(Security) --> A[Discover]
D(Any app) --> A[Discover]

Loading

This works although creates a direct dependency on Discover from any consumer.

As this registry doesn't rely on particular lifecycle hooks, we could factor it out into a package which works as an IoC container for Discover features, which would make the dependency tree look something like:

flowchart TD

A(Discover) -- Get --> E[DiscoverFeaturesRegistry]
B(Logs Explorer) -- Set --> E[DiscoverFeaturesRegistry]
C(Security) -- Set --> E[DiscoverFeaturesRegistry]
D(Any app) -- Set --> E[DiscoverFeaturesRegistry]

Loading

It would so reflect the decisions taken in https://github.com/elastic/kibana/pull/166524/files?short_path=569daf5#diff-569daf56743cc51d98ac5239825483015505fdce810a0afaa6dd353e1d3caade and reduce the coupling between dependencies.
Details need to be defined, although some experimental work will start on the first part of this migration.

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)