Should we remove file-based scripts and templates?

[clintongormley]

We have largely removed support for file-based configuration (eg mappings, settings, index templates) in favour of storing this info in the cluster.

For scripts and mustache templates, we allow them to be passed inline or to be stored in the cluster state. We used to be concerned about locking down scripts in Groovy etc, but with Painless we have safe scripting. Are file based scripts now redundant?

The one upside of files is that scripts can be written with new lines and formatting, while with inline/stored scripts they have to be passed in in a single line of JSON with newlines escaped. However this could be solved by improving Console to do the multi->single line conversion automatically.

Any other reasons for keeping file scripts?

[rjernst]

See #10959

[dadoonet]

Totally agreed. People should use API to distribute their scripts in the cluster.
I also agree that improving Console to generated a single line version of a script is important to have before we definitely remove that feature.
It can be deprecated in 5.x IMO.

[clintongormley]

I'm leaving this open for feedback from users - are there situations solved by having file based scripts which are not solved by stored scripts?

[tlrx]

It know it is often used to create scripts with automated deployment tools like ansible or chef. These tools could use the REST API too but it is usually easier to copy files rather than waiting for the cluster to be green/put the script/handle retries and authentication.

We could maybe use a "drop in" folder where the user can copy the scripts and they will be loaded and added as stored script by the node itself? This would still allow to deploy scripts as simple files, and plugins during their installation could also add files in this drop in folder (and then they won't need to access the filesystem). Once processed/acked, file can be deleted by the node.

[clintongormley]

We could maybe use a "drop in" folder where the user can copy the scripts and they will be loaded and added as stored script by the node itself?

The way to do this is, when building your package, to fire up a one node cluster with the right cluster name, add all the scripts/templates/whatever that is needed for the new cluster, then shut the node down and use it as a base image.

Then when deploying, the first node in your cluster has all the right settings etc, and the other nodes that join the cluster inherit them directly from the cluster state (ignoring whatever they have locally).

[bleskes]

We discussed this further on fix it friday. We are tending to remove file based scripts but want to wait more and get more feedback from users.

Some finer points of the discussion that are worth mentioning:

1. File base scripts are used today to add scripts while blocking them completely in the APIs. Even if we remove file based scripts people can still use plugins for this case.
2. The method suggested by @clintongormley works for the initial deployment of a cluster, it does however fail when one want to modify an existing cluster. For example, currently you can shutdown the cluster, update the scripts, start the cluster and have new scripts deployed.

[javanna]

Seems like we got no feedback from users in 5 months. Maybe time to mark adoptme?

[clintongormley]

I'm in favour of removing file scripts and just keeping stored and inline. The APIs for storing scripts should be protected by the application. Marking as adoptme