Changing cross cluster search settings using REST API

[redlus]

Hi,

Elasticsearch version
6.2.3

Plugins installed
ingest-attachment
ingest-geoip
mapper-murmur3
mapper-size
repository-azure
repository-gcs
repository-s3

JVM version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior
We're using the cross-cluster search to make searches across several clusters. The settings are pushed using PUT to _cluster/settings on the live clusters. When acknowledged, the remote clusters get the default settings as views in _remote/info.

Trying to change these settings through the same API raises errors; for example, calling:
{ "persistent": { "search": { "remote": { "tier-2": { "seeds": "192.168.0.47:9300", "initial_connect_timeout": "60s" }, "tier-1": { "seeds": "192.168.0.45:9300", "initial_connect_timeout": "60s" } } } } }

returns this:
{ "error": { "root_cause": [ { "type": "remote_transport_exception", "reason": "[elasticmaster1][192.168.0.42:9300][cluster:admin/settings/update]" } ], "type": "illegal_argument_exception", "reason": "persistent setting [search.remote.tier-1.initial_connect_timeout], not recognized" }, "status": 400 }

If these settings should be changed only in elasticsearch.yml, it should be stated so in the documentation. However, given that new remote clusters can be set on the fly, I see not reason to support different settings only though one option and not the other.

Steps to reproduce

1. Create two clusters (can be one node each).
2. Introduce two aliases to one of the cluster, one alias is its own, the other is for searching the other cluster. Add one of the settings (aside from 'seed') to the PUT call.
3. Eat your popcorn and enjoy the error message.

Provide logs (if relevant)
The master node presents the following log message:

[2018-05-27T12:11:37,841][DEBUG][o.e.a.a.c.s.TransportClusterUpdateSettingsAction] [elasticmaster1] failed to perform [cluster_update_settings]
java.lang.IllegalArgumentException: persistent setting [search.remote.tier-1.initial_connect_timeout], not recognized
at org.elasticsearch.common.settings.AbstractScopedSettings.updateSettings(AbstractScopedSettings.java:574) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.common.settings.AbstractScopedSettings.updateDynamicSettings(AbstractScopedSettings.java:520) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.action.admin.cluster.settings.SettingsUpdater.updateSettings(SettingsUpdater.java:60) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.action.admin.cluster.settings.TransportClusterUpdateSettingsAction$1.execute(TransportClusterUpdateSettingsAction.java:183) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:273) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) [elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) [elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:566) [elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.2.0.jar:6.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

[jasontedor]

"type": "illegal_argument_exception", "reason": "persistent setting [search.remote.tier-1.initial_connect_timeout], not recognized"

The problem here is that there is no per remote-cluster setting called intial_connect_timeout. You will get a similar error message if you try to add such a setting to the configuration file:

Caused by: java.lang.IllegalArgumentException: unknown setting [search.remote.tier-1.initial_connect_timeout] did you mean [search.remote.initial_connect_timeout]?
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:344) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:308) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:282) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:135) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
	... 6 more

If you think the docs are not clear on this point, we are open to the feedback, yet contrast the documentation for search.remote.initial_connect_timeout versus search.remote.${cluster_alias}.skip_unavailable with the latter indicating that it is a per remote-cluster setting.

Do note that this setting is indeed not dynamic however I think it is fine as a static setting, given that it is not per remote-cluster.

Eat your popcorn and enjoy the error message.

Comments like this are unhelpful and unnecessary.

[redlus]

@jasontedor
I've only written the requests and response when inserting the setting at the cluster level, but this happens on every level. E.g. on 'search.remote' level:
{ "persistent": { "search": { "remote": { "tier-2": { "seeds": "192.168.0.47:9300" }, "tier-1": { "seeds": "192.168.0.45:9300" }, "initial_connect_timeout": "60s" } } } }

Response:
{ "error": { "root_cause": [ { "type": "remote_transport_exception", "reason": "[elasticmaster1][192.168.0.42:9300][cluster:admin/settings/update]" } ], "type": "illegal_argument_exception", "reason": "persistent setting [search.remote.initial_connect_timeout], not dynamically updateable" }, "status": 400 }

On 'search' level:
{ "persistent": { "search": { "remote": { "tier-2": { "seeds": "192.168.0.47:9300" }, "tier-1": { "seeds": "192.168.0.45:9300" } }, "initial_connect_timeout": "60s" } } }

Response:
{ "error": { "root_cause": [ { "type": "remote_transport_exception", "reason": "[elasticmaster1][192.168.0.42:9300][cluster:admin/settings/update]" } ], "type": "illegal_argument_exception", "reason": "persistent setting [search.initial_connect_timeout], not recognized" }, "status": 400 }

[jasontedor]

As I said:

Do note that this setting is indeed not dynamic however I think it is fine as a static setting, given that it is not per remote-cluster.

You have to set this statically in the configuration file.

[redlus]

@jasontedor
I felt the emphasis was needed after your reference of what is and what is not a per-cluster setting:

contrast the documentation for search.remote.initial_connect_timeout versus search.remote.${cluster_alias}.skip_unavailable with the latter indicating that it is a per remote-cluster setting.

In any case, stating all these settings in one section without noting which is static and which is not is confusing IMHO (especially given that the cluster setting API is mentioned as an alternative to the elasticsearch.yml option) -

The equivalent example using the cluster settings API to add remote clusters to all nodes in the cluster would look like the following

More importantly, I hold the belief that almost any setting which can become dynamic should definitely have this option. Too many times we find outselves needing to perform a rolling restart of large elasticsearch clusters just to activate a feature or fine-tune one that has already been activated using a rolling-restart. This time I'm referring to this issue (and setting) - #30687.