[config] mechanism to prevent internal configs from leaking to self-managed users

[lukeelmers]

We plan to use serverless config files to manage static settings for each project type we want to support in our serverless offering. However, the nature of our existing yml config options are that they're essentially a public API. There is no way to differentiate between settings that are meant to be adjusted by users, vs ones that are not. A good example here are our various cloud settings that are used in ESS: technically these could be set by self-managed users. In the case of cloud, these are pretty useless/harmless settings outside of ESS, but there are more serious quality/testing concerns when we think of the types of config options we'll likely be adding for serverless.

We will ultimately have tests running against each project type for the given combination of configs representing that type. However, there is a risk in some of these configs being picked up by self-managed users as it would be possible to run Kibana in unsupported/untested configurations. This is particularly a concern with the special behavior of ${plugin}.enabled settings; we do not plan to have tests for every possible combination of plugins.

The best way to minimize risk of these added configurations is to design a mechanism where we can identify specific configs as internal, and then somehow exclude them if Kibana is not running in serverless. Potentially this solution could also be used for ESS-only settings, though that isn't a hard requirement at this time.

One idea that was floated was investigating whether we could introduce another build specifically for serverless, and then conditionally include these settings there so that they would otherwise be excluded in the public distributable (cc @elastic/kibana-operations). This may be something we want to consider regardless as there are plans to build some serverless-only plugins.

If we don't introduce a safety measure like this, we would need to explicitly document these settings as not supported... as we have seen in the past, simply leaving them undocumented doesn't mean they won't be used. 🙂

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

Concretely what we want here is a mechanism to disable/enable some plugins based on the project type without exposing said mechanism as a public API (such as kibana.yml).

The solution that seems like it would require the least amount of work in Core to achieve this would be to create a serverless build and then use schema.conditional(schema.contextRef('serverless')... to allow some plugins to be disabled in the serverless build. The config files can then drive the exact set of plugins that are enabled/disabled per project type so that we don't need a new build per project type.

---

We plan to use #150616 to manage static settings for each project type we want to support in our serverless offering.

This is somewhat of an aside... While this mechanism allows any per project configs, in general I'd prefer if we don't introduce any new config flags just for the ability to specify them in these per project config files. Rather the config files should use existing config properties or only enabling/disabling plugins. Further customisation would then be powered through plugin JS APIs

If observability wants to add a new embeddable type to their project type, we could add a config flag like enableObservabilityCustomEmbeddable but it would be preferable if some observability plugin or serverless.observability plugin registers the custom embeddables it wants. This plugin would then be enabled for observability project types but not security ones. Without knowing the exact nature of the adjustments we want to make it's hard to be confident that one is better than the other, but I would lean towards reducing the flexibility and maintenance burden until the needs are validated. If there is already an existing config flag to achieve the desired tweaking then there's no additional maintenance burden and setting it in the per-project config file is therefore not an issue.

[pgayvallet]

Concretely what we want here is a mechanism to disable/enable some plugins based on the project type without exposing said mechanism as a public API

Are we certain this is all this issue is about (I would be very glad if it was, but still asking)? From @lukeelmers's initial discussion, it felt like it could be more than just having a 'serverless only' way to control plugin enablement, and could be the need to have specific config properties (either Core's or Plugins') having the same need?

[rudolf]

it felt like it could be more than just having a 'serverless only' way to control plugin enablement, and could be the need to have specific config properties (either Core's or Plugins') having the same need?

Yeah I think that's very much still in the air. But once we have a serverless build we can use the same mechanism to only expose some config flags to serverless if we want these serverless-specific configs to be a private API.

So if we want to customize the help button link with serverless.project_type_specific_help_button_link: 'https://elastic.co/security' then we'd wrap that in schema.conditional(schema.contextRef('serverless')... to disable it for on-prem.

I think we should be careful because of the maintenance burden, but other than that this seems sufficient at the technical level.

[rudolf]

After looking through the code I realise I misunderstood the schema.contextRef API. We don't need a serverless build, we just need to add a boolean serverless property to the Env. Since #150616 already introduced a --serverless CLI flag we could just set the Env flag if this CLI arg was used.

[pgayvallet]

FWIW, this is where we're adding the context usable by references when validating the config:

kibana/packages/kbn-config/src/config_service.ts

Lines 239 to 247 in 1cbf83f

	return schema.validate(
	config,
	{
	dev: this.env.mode.dev,
	prod: this.env.mode.prod,
	...this.env.packageInfo,
	},
	`config validation of [${namespace}]`
	);

The CLI command reading the serverless flag:

kibana/src/cli/serve/serve.js

Line 262 in 065dbe7

const serverlessMode = getServerlessProjectMode(opts);

However, atm this flag is is not injected into CliArgs or EnvOptions passed to Core's Root, so we will need to do that too.

[afharo]

This issue might be related: #155912