[Knowledge Base] Make the elastic.co doc accessible to our AI assistants

[legrego]

We want to make our product documentation available to our Kibana AI assistants, so that they can retrieve and utilize this knowledge when performing their tasks, greatly improving their performances in scenarios where they need to retrieve knowledge about our products to answer specific questions.

We plan to install these resources via a Fleet integration, since this is a well-tested model for distributing assets, even in air-gapped environments.

Once installed, this feature will be exposed to solution teams via an LLM task exposed from the inference plugin. This approach allows the AppEx teams to keep the internals (how we retrieve the doc, where it’s stored and so on) an implementation detail in case we need to change it later (e.g. if we want the feature as a global external service at some point).

[elasticmachine]

Pinging @elastic/appex-ai-infra (Team:AI Infra)

[pgayvallet]

So after discussions with the group, the consensus is now that it's not worth following the approach for the MVP (which was shipping a subset of the documentation in the Kibana distributable) and that we should focus on the long term plan instead, so using fleet packages as the shipping and installation mechanism for our KB artifacts.

This plan can be split into 4 main tasks:

1. ELSER / interface endpoint installation and management #192461 (which is a prerequisite for the current issue)
2. [KB] add KB support to fleet package spec and implementation #193849
3. [KB] Add documentation packaging workflow #193473
4. [KB] expose documentation retrieval as LLM task #193474

1. and 2. can be started in parallel.
3. kinda depends on 2., as we need to know how we will be building the fleet package
4. depends on all the previous steps

[pgayvallet]

PR has been merged, this is considered completed.