Coordinator nodes cannot be defined using node.roles

[charith-elastic]

This was mentioned in #3324 but is worth raising as a separate issue.

ucfg seems to treat empty arrays as null values so the suggested method of defining a coordinator node cannot be done on ECK by setting node.roles: [ ].

ElasticsearchParseException[null-valued setting found for key [node.roles] found at line number [16], column number [10]]

[charith-elastic]

Workaround is to use YAML type tags:

node.roles: !!seq ""

This is probably not going to be obvious to the users and slightly deviates from the Elasticsearch documentation (even though they are semantically equivalent.)

Other options that I could think of are:

• Make config a strongly-typed struct to help ucfg figure out how to deserialize each field.
  • Con: we'll have to keep on top of any changes to the Elasticsearch configuration schema.
• Ask the Elasticsearch team to consider adding coordinator as a valid role name.
  • Con: as the empty array is already an accepted value in a released version, they will have to keep supporting it as well but ECK won't be able to.
• Ditch ucfg and use standard YAML parsing libraries
  • Con: Handling dot separated setting keys is going to be tricky.
• Ask ucfg to support external schemas/type hints for parsing certain paths.

[charith-elastic]

Added a note to docs/orchestrating-elastic-stack-applications/elasticsearch/node-configuration.asciidoc in #3719 about the YAML type tag workaround. If a different solution is found, that note should be removed.

[pebrc]

I don't think the workaround works as expected. On 7.10.0 with node.roles: !!seq "" all node roles seem to be configured. The rendered configuration on the node has node.roles: ""

10.73.50.49 41 65  4 1.21 0.41 0.27 cdhilmrstw - elasticsearch-es-coordinating-0

[pebrc]

The only workaround <8.0 that I could find is to use the legacy node role format as in:

  - config:
      node.master: false
      node.data: false
      node.ingest: false
      node.ml: false
      node.remote_cluster_client: false

which results in

10.131.0.126 11 63  6 0.38 0.63 0.53 -          - elasticsearch-es-coordinating-0

[pebrc]

The underlying problem is that once we create ucfg Config object out of the the YAML we lose the type information that node.roles was pointing to a empty slice of strings and instead we end up with a nil valued interface{}. Standard go.yaml renders the former as [] and the latter as null.

[pebrc]

#4041 implements a workaround. I am leaving this issue open for a proper solution.

[bvierra]

Using eck-operator 1.5 with ES 7.12.0 with the following:

node.roles: !!seq ""

is accepted via the operator, however all of the roles were enabled on the node.

Using:

- config:
      node.master: false
      node.data: false
      node.ingest: false
      node.ml: false
      node.remote_cluster_client: false

worked as expected

[sebgl]

In case it was not clear: starting ECK 1.4.0 this bug is solved. You can specify node.roles: [] to indicate you want a coordinating-only node. It should work fine.

This issue is left open because we are not satisfied with how we implemented that fix :)

[jen-huang]

Hi, we had a report of the same issue coming up for a Kibana YAML setting: #4853

I will leave it to this team to consolidate these two issues as needed :)

[thbkrkr]

I will check that we can revert #4041 with elastic/go-ucfg#188.

[pebrc]

I will check that we can revert #4041 with elastic/go-ucfg#188.

Yes 💯But we should keep a unit test around that verifies that the configuration with empty array values works.

[piyushkp]

i'm upgrading from ES 8.2.0 to 8.4.0 and coordinator nodes failing with: (it worked fine in 8.2.0)

Exception in thread "main" org.elasticsearch.ElasticsearchParseException: null-valued setting found for key [node.roles] found at line number [9], column number [12] at org.elasticsearch.common.settings.Settings.validateValue(Settings.java:768) at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:743) at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:687) at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1190) at org.elasticsearch.node.InternalSettingsPreparer.loadOverrides(InternalSettingsPreparer.java:143) at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:53) at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:110) at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85) at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:94) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85) at org.elasticsearch.cli.Command.main(Command.java:50) at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)

```
    roles: []
    # For client nodes, we also need to add an empty node.roles in elasticsearch.yml
    # This is due to https://github.com/elastic/helm-charts/pull/1186#discussion_r631225687
    esConfig:
      elasticsearch.yml: |
        node.roles: []
```