[WIP] Operator and plugin read-only settings

[grcevski]

Introduction

This PR introduces the concept of operator settings. The operator setting have the following properties:

• They are read-only settings that cannot be changed by the REST API.
• They can be cluster settings or entity settings like ILM policies.
• The operator settings can be created/modified or deleted only through the 'operator mode'. The only operator mode available in this PR is file based settings, however the intent is to enable plugins and modules to set their own 'immutable' settings.

Implementation details

We have multiple components introduced by this PR:

File Settings Service

Watches a directory under Elasticsearch config for changes to a file called settings.json. This is a JSON file describing the settings that the operator wants to set that will be immutable through the REST API. Any changes to these settings will only be allowed through modifying the file. As configured by default we watch for a path operator/settings.json under the config directory. The name of the directory operator can be changed via settings.

The File Settings Service will notice updates to the file and perform an update. The operator directory doesn't have to exist, the file setting service will notice when it's created. Recommended way to change operator file based settings is through modifying the settings.json file in a brand new location and then performing a symlink under the Elasticsearch config directory.

Operator Cluster State Controller

This is a reusable component that has number of Operator Handler components that know how to update parts of the Elasticsearch state. This component is only used by the File Settings Service at the moment, but the intent is for modules and plugins to be able to call it to save settings and entities in the cluster state that cannot be changed by the end user.

Operator Metadata

This is an extension to the ClusterState Metadata object where we store various information about the operator updated state. This Operator Metadata object contains information about the version and the 'keys' that were set in the cluster state, through the operator interface. We use this information to coordinate updates and to ensure that the keys of the cluster state set by the operator are not overwritten by the REST API.

Example operator state to be set by file based settings

{
    "metadata": {
        "version": "1234",
        "compatibility": "8.4.0"
    },
    "state": {
        "cluster_settings": {
            "indices.recovery.max_bytes_per_sec": "50mb"
        },
        "ilm": {
            "my_timeseries_lifecycle": {
                "policy": {
                    "phases": {
                        "warm": {
                            "min_age": "10s",
                            "actions": {
                            }
                        },
                        "delete": {
                            "min_age": "30s",
                            "actions": {
                            }
                        }
                    }
                }
            }
        }
    }
}

In the example above we can see the following details:

Metadata information

We show an example of how the version can be set along with the Elasticsearch Version for compatibility. The version metadata field is assumed to be of type long and it's encoded as a JSON String to avoid precision errors in certain JSON language implementations (e.g. JavaScript). For proper usage, each time the cluster state is updated we need to bump the version. The version bump has to be increasing in value, but not necessarily monotonically increasing, e.g. we can use a consistent epoch timestamp generator for versioning purposes.

The compatibility field is used to enforce a minimum Elasticsearch version that can understand the file content. We can use this field to avoid trying to apply the cluster settings to an incompatible Elasticsearch version in mixed version clusters.

State information

In this case we show an example of two separate operator state handlers: cluster settings and ilm.

This cluster state information will be applied as one single cluster state update. Even though the two transport actions are separate when we use the REST API, with operator state updates we first validate everything up-front and then we use a joint cluster state update.

In the example above, the final OperatorMetadata will contain information about two OperatorHandlers, cluster_settings and ilm, which individually will contain the keys of the settings that are updated (indices.recovery.max_bytes_per_sec and my_timeseries_lifecycle respectively). If these two settings were set as an operator state, any REST API requests that want to modify indices.recovery.max_bytes_per_sec or mutate/delete the my_timeseries_lifecycle policy, would be rejected.

Scope of the PR

This PR doesn't handle yet updates to role mappings which are not stored in the cluster state. Given the PR is large as-is, the ability to set role mappings through in operator mode will be added as a follow-up PR.

Documentation will also be done as separate PR.

This PR will be used to finalize the file based settings changes after the following sub PRs are finished:

• Metadata classes for Operator state (Add immutable 'operator' metadata classes for cluster state #87763)
• OperatorHandler and its SPI (Add OperatorHandler interface #87767)

TODO

• Unit tests for everything
• Integrations tests
• Docker tests
• Use SPI instead of extending the plugin interface

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[grcevski]

@elasticmachine run elasticsearch-ci/part-2

[grcevski]

Closing this Draft PR, all of the changes are either in or in waiting on PR review.