Eliminate APIs that depend on local files

[rjernst]

We have a number of APIs that rely on the user placing files on every node. For example:

• File scripts must exist in the scripts dir on every node
• Some analyzer filters allow to specify their settings in local files, like a stopwords list or synonyms rules

Requiring to place a file on every node is error prone, and this issue is to discuss how we can lock down all APIs like this, and track the effort.

Last year, we added "indexed scripts", which is a special .script index that exists on every node. The index is very simple: it just stores the contents of the script, and can be looked up by the "document" id. I wonder if we could generalize this to a "resources" index, which could have scripts, token filter setup, and anything else we might need in the future. Even plugins could be forced to use this, so we could remove the ability for plugins to read from local disk altogether? As far as security, I think the resources index should have the ability to be "locked" (metadata block?), which is essentially the same as disabling updating for indexed scripts today.

[clintongormley]

It would be awesome to centralise resource management like this. A couple of issues:

• we'd have to make sure that the "resources" index recovers before all other indices, because they may depend on it (eg for analysis setup)
• corruption or unavailability of this index could impact the rest of your cluster
• some plugins need to be setup before the cluster has started (eg discovery) (but typically these don't require resources other than config)
• how do you lock the resources index without also providing an API to unlock it?

[javanna]

How could we address scripting languages that don't support sandboxing? I guess locking/unlocking the resources index should not be exposed via api, otherwise anybody can use it and do whatever they can? Or am I missing something?

[rjernst]

how do you lock the resources index without also providing an API to unlock it?

How could we address scripting languages that don't support sandboxing? I guess locking/unlocking the resources index should not be exposed via api, otherwise anybody can use it and do whatever they can?

I don't think this is any different than locking APIs like creating indexes or updating mappings. The "resources" api should be able to have a block on modifications. This is the same as e.g. blocking other metadata actions (and yes I view this as metadata, even though the underlying impl would be in a special index. I'm not sure what indexes scripts are classified as today).

[tsullivan]

Would this possibly alleviate the current pain point requirement to close and re-open the index to re-read the synonym database?

[garyui]

I agree - closing and reopening the index , or restarting each node in the cluster, one by one, is very painful. Anything that you can do to relieve this painful process would be well received!

[babadofar]

+1

[rjernst]

I'm closing this issue because the original idea (using an index) has many issues, and there are other current ideas which will improve the situation eg for synonyms. The idea of "don't use local files" is well engrained in the culture now and they are used sparingly. Additionally, file based scripts are now gone.