Files plugin - swap out "get by ID" calls to make it compatible with index aliases

[hop-dev]

Over on Fleet we have used the file plugin for upload and download of agent diagnostic files, we use ILM to manage the file data and metadata indices, and therefore have to use an index alias infront of the backing indices. The problem we have encountered is that after a rollover, the index alias points to multiple indices, which means get doc by ID calls fail with:

Root causes:
		illegal_argument_exception: alias [.fleet-files-endpoint] has more than one index associated with it [.fleet-files-endpoint-000001, .fleet-files-endpoint-000002], can't execute a single index op

This is because get by ID is a single index op.

We want to understand how big a change and how much risk there would be in moving to using _search to find by ID instead of the get by ID API. This is currently blocking 8.7 so we need to know if there is a realistic way forward.

Here is the fleet bug we raised #153322

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/appex-sharedux (Team:SharedUX)

[vadimkibana]

From what I understand, this line of code is reading file chunks by get ID, but since you are not pointing the read stream to an index, but an alias, which points to multiple indices, it fails.

It seems like a quick and dirty fix could be to replace that line with _search by ID.

Retrieving using get by ID is more efficient. So, maybe a better solution would be to leave the get by ID logic by default, and add the ability to fetch by _search if somebody opts-in. For example, by specifying a flag to the ContentStream constructor:

new ContentStream({
  indexIsAlias: true, // or fetchBySearch: true
});

[hop-dev]

I think as _search will work for both index and alias, and the performance impact shouldn't be noticeable, I would lean towards having one code path instead of the flag to save some complexity. But of course I am happy to take your direction.

I am on call at the minute so need to tend to a couple of SDHs, I'm then happy to look at getting a PR together.

[vadimkibana]

[..] the performance impact shouldn't be noticeable [..]

It might be noticeable. Found this answer on the Wiki:

From there:

• get by ID is automatically routed to the correct shard, whereas search request is fanned out to all shards.
• The _search query needs to be rewritten as a bool query and search mechanics need to be used, instead of just fetching a document by its ID.
• Using get the latest document is returned, whereas, when using _search, it needs to go through the Lucene search index, and where the document will only be available after a "refresh", which is 1s by default. And I believe, Elasticsearch Serverless was aiming for 5s refresh interval. (If that is true, files downloaded with get will be immediately available, but if "refresh" interval is 5s and _search is used—the already uploaded files will only be possible to download after 5s).

[kpollich]

I'm +1 for implementing this via an opt-in flag to use the _search API and leaving the default behavior as fetching via get. This seems like the least breaking way to handle this, and allows us to introduce the new behavior in only the code paths experiencing issues (Fleet + Security Solution UI file uploads).

Let's make sure we include a comment regarding the new option that explains a consumer may want to use the _search behavior if they're reading from an alias.

[paul-tavares]

Hi all,
I'll start to look at implementing the suggested approach mentioned above by adding a new argument to the exposed service that will make the internals of the service use _search instead. I'm still not clear on the extent of the changes, but I'll dig in now

[paul-tavares]

Make the initial set of changes here:

#153342

It gets the Security Solution use case working again and I think it also addresses Fleet's use case - still testing. If someone can take a look at the changes thus far, it would be much appreciated