[FEATURE]: Kibana to watch certificate changes and reload them on-the-fly

[danlsgiga]

Describe the feature:
Kibana should reload certificates on-the-fly if the certificate files are renewed / replaced. Right now it requires a full restart to reload certificates.

Describe a specific use case for the feature:
We have a very dynamic environment where certificates are short lived and managed by Vault. Every time a certificate is renewed and/or replaced by Vault it requires a full Kibana restart. I've tried doing a SIGHUP and SIGUSR1 in the kibana process to force a certificate reload but that didn't help.

Note: Elasticsearch is able to reload certificates on-the-fly already so I just assumed Kibana would be able to do the same.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[tylersmalley]

SIGHUP should reload the certificate - if it's not, that is a bug.

[danlsgiga]

I'm running Kibana 7.5.1 and a SIGHUP does not reload the certs. Here is the error after the SIGHUP.

{"type":"error","@timestamp":"2020-01-08T21:41:49Z","tags":["connection","client","error"],"pid":10585,"level":"error","error":{"message":"140388375709568:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 42\n","name":"Error","stack":"Error: 140388375709568:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 42\n"},"message":"140388375709568:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 42\n"}

[J7mbo]

It shoudn't need a SIGx though. Ideally Kibana would do the same as elasticsearch and reload automatically when the certs change.

[danlsgiga]

Agreed, and that was my expectation but it's not the case. Kibana is not detecting certificate changes and consequently not reloading them. Currently I need a full restart to get them updated.

[J7mbo]

Hacky workaround until it's implemented, (if we're lucky enough!): use inotifywait to watch for your certificates changing and then kill / restart kibana or use the SIGHUP if it works.

[danlsgiga]

That's what I have in place atm, so once a month (we have short lived certs) we do a full restart to reload the cert. SIGHUP or SIGUSR1 doesn't work!

[Funbit]

I can't understand why this ticket hasn't been resolved yet... Everybody who uses LetsEncrypt to update the certificates faces with this reloading issue. Taking into account that Elasticsearch instance is already capable of on-the-fly reloading (and all other major open source servers I know), it should be a piece of cake to fix.
Please, make the fix happen 🥺 🙏

[cataclysdom]

We're trying to move to automating Let's Encrypt certificate renewals since most browsers have announced their intent to limit the lifetime of certificates. However, rebooting Kibana once a month is not an option.

Read more here: Mozilla Joins Apple, Google in Reducing TLS Certificate Lifespans.

[sastorsl]

This is affecting us all, and more and more, since the whole industry is moving towards shorter lifetimes on certificates (not just Let's Encrypt).

Either a way for kibana to automatically reload certificates, or at least a trigger which starts a restart (optional of course).

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[legrego]

Possible duplicate of #101072

Duplicated by #101072

[Flixi555]

Any updates on this? Even with two year certificates this is becoming a real issue for us.

[pestevao]

Is a real issue for everyone who uses Kibana, Elastic and so on with all security recommendations in place.
It's incomprehensible that https and encryption is (and well!) recommended everywhere on ELK system but... since 2020 - two years if not earlier - to implement a mandatory change in order to prevent restarts on productive systems. And like Elastic already do it for long time. So it's not a "rocket science" request, seems.

[pgayvallet]

So it's not a "rocket science" request, seems.

PRs are always welcome @pestevao

[loganmzz]

Still restart required or at least signals are fixed ? Find nothing in documentation about one or the other.

[nicmr]

@pgayvallet since you mentioned PRs are welcome, could you give directions on the places in the codebase that would need to be touched for such a contribution? That would make it easier for new contributors

[pgayvallet]

Sure thing, my apologies for not having done it sooner.

First notable information, HAPI is the underlying HTTP library we're using on Kibana.

Technical pointers

From config definition to consumption to utilization to create the HAPI server:

SSL config definition

This is the server.ssl schema definition:

kibana/packages/kbn-server-http-tools/src/ssl/ssl_config.ts

Lines 22 to 78 in 265cc76

	export const sslSchema = schema.object(
	{
	certificate: schema.maybe(schema.string()),
	certificateAuthorities: schema.maybe(
	schema.oneOf([schema.arrayOf(schema.string()), schema.string()])
	),
	cipherSuites: schema.arrayOf(schema.string(), {
	defaultValue: cryptoConstants.defaultCoreCipherList.split(':'),
	}),
	enabled: schema.boolean({
	defaultValue: false,
	}),
	key: schema.maybe(schema.string()),
	keyPassphrase: schema.maybe(schema.string()),
	keystore: schema.object({
	path: schema.maybe(schema.string()),
	password: schema.maybe(schema.string()),
	}),
	truststore: schema.object({
	path: schema.maybe(schema.string()),
	password: schema.maybe(schema.string()),
	}),
	redirectHttpFromPort: schema.maybe(schema.number()),
	supportedProtocols: schema.arrayOf(
	schema.oneOf([
	schema.literal('TLSv1'),
	schema.literal('TLSv1.1'),
	schema.literal('TLSv1.2'),
	schema.literal('TLSv1.3'),
	]),
	{ defaultValue: ['TLSv1.1', 'TLSv1.2', 'TLSv1.3'], minSize: 1 }
	),
	clientAuthentication: schema.oneOf(
	[schema.literal('none'), schema.literal('optional'), schema.literal('required')],
	{ defaultValue: 'none' }
	),
	},
	{
	validate: (ssl) => {
	if (ssl.key && ssl.keystore.path) {
	return 'cannot use [key] when [keystore.path] is specified';
	}
	if (ssl.certificate && ssl.keystore.path) {
	return 'cannot use [certificate] when [keystore.path] is specified';
	}
	if (ssl.enabled && (!ssl.key || !ssl.certificate) && !ssl.keystore.path) {
	return 'must specify [certificate] and [key] -- or [keystore.path] -- when ssl is enabled';
	}
	if (!ssl.enabled && ssl.clientAuthentication !== 'none') {
	return 'must enable ssl to use [clientAuthentication]';
	}
	},
	}
	);

which is then used when building the global http configuration used by our service:

kibana/packages/core/http/core-http-server-internal/src/http_config.ts

Line 320 in 8727c68

this.ssl = new SslConfig(rawHttpConfig.ssl || {});

kibana/packages/kbn-server-http-tools/src/ssl/ssl_config.ts

Lines 82 to 85 in 265cc76

	export class SslConfig {
	public enabled: boolean;
	public redirectHttpFromPort?: number;
	public key?: string;

Config creation / consumption

The http configuration is then instantiated in our HttpService, here:

kibana/packages/core/http/core-http-server-internal/src/http_service.ts

Lines 73 to 77 in 2aa8e6d

	this.config$ = combineLatest([
	configService.atPath<HttpConfigType>(httpConfig.path),
	configService.atPath<CspConfigType>(cspConfig.path),
	configService.atPath<ExternalUrlConfigType>(externalUrlConfig.path),
	]).pipe(map(([http, csp, externalUrl]) => new HttpConfig(http, csp, externalUrl)));

Note that atm, even if the config is an observable (therefore can be hot-reloaded via SIGUP), the service doesn't leverage this feature and only uses the first emission (so the config as it was when the process started):

kibana/packages/core/http/core-http-server-internal/src/http_service.ts

Line 158 in 2aa8e6d

const config = await firstValueFrom(this.config$);

kibana/packages/core/http/core-http-server-internal/src/http_service.ts

Lines 160 to 163 in 2aa8e6d

	const { registerRouter, ...serverContract } = await this.httpServer.setup(
	config,
	deps.executionContext
	);

Which is one of the things that will have to change.

Conversion to HAPI primitive

Our configuration is then converted to its HAPI version to be able to pass it down to the underlying HAPI server instantiation:

kibana/packages/core/http/core-http-server-internal/src/http_server.ts

Lines 216 to 219 in 8727c68

	const serverOptions = getServerOptions(config);
	const listenerOptions = getListenerOptions(config);
	this.config = config;
	this.server = createServer(serverOptions, listenerOptions);

The SSL config conversion, in particular, is done within getServerOptions, here:

kibana/packages/kbn-server-http-tools/src/get_server_options.ts

Lines 53 to 72 in 265cc76

	if (configureTLS && config.ssl.enabled) {
	const ssl = config.ssl;
	// TODO: Hapi types have a typo in `tls` property type definition: `https.RequestOptions` is used instead of
	// `https.ServerOptions`, and `honorCipherOrder` isn't presented in `https.RequestOptions`.
	const tlsOptions: TLSOptions = {
	ca: ssl.certificateAuthorities,
	cert: ssl.certificate,
	ciphers: config.ssl.cipherSuites?.join(':'),
	// We use the server's cipher order rather than the client's to prevent the BEAST attack.
	honorCipherOrder: true,
	key: ssl.key,
	passphrase: ssl.keyPassphrase,
	secureOptions: ssl.getSecureOptions ? ssl.getSecureOptions() : undefined,
	requestCert: ssl.requestCert,
	rejectUnauthorized: ssl.rejectUnauthorized,
	};
	options.tls = tlsOptions;
	}

Starting the HAPI server

The underlying HAPI server is then started during HttpServer.start()

kibana/packages/core/http/core-http-server-internal/src/http_server.ts

Lines 289 to 295 in 8727c68

	await this.server.start();
	const serverPath =
	this.config && this.config.rewriteBasePath && this.config.basePath !== undefined
	? this.config.basePath
	: '';
	this.log.info(`http server running at ${this.server.info.uri}${serverPath}`);

What must be done

1. Having the HttpServer be aware of SSL config changes

There could be two approaches here:

• a. use the SIGHUP-based config hot reload mechanism already in place
• b. automatically detect changes in certificate files without the need of SIGHUP

Option a is strongly encouraged, and I will assume this is the one we go with. We can't provide insight or pointers for option b, given there's nothing in Kibana functioning that way today.

The setup signature of HttpServer will have to change, to receive the config observable config$ instead of the config object itself:

kibana/packages/core/http/core-http-server-internal/src/http_service.ts

Lines 160 to 163 in 2aa8e6d

	const { registerRouter, ...serverContract } = await this.httpServer.setup(
	config,
	deps.executionContext
	);

That way, the HttpServer will be able to subscribe to configuration changes.

2. Having the HttpServer react to config changes

That's the tricky part. Here to, we have identified two options (but there might be more):

• a. have the underlying HAPI server hot-reload the certificate changes
• b. fully recreate the server when the certificate changes

a. Hot reloading the underlying HAPI server

recommended

Last time we checked, HAPI doesn't support hot reloading natively. However, it should still be possible to access the raw http/tls server it creates under the hood to use either setSecureContext or SNICallback.

HAPI.Server has a listener: http.Server property that should be a tls.Server when tls is configured.

This would be our recommended approach: find a way to use either of those low-level APIs to perform the certificate hot-reloading operation.

b. Fully recreating the server

discouraged

The alternative would be to fully stop and recreate the server when a config change is encountered. This would be significantly more complex though, given I think we would need to fully reinstantiate a new HAPI.server instance, which would break the assumption we have within the http domain layer that this instance is not replaceable.

E.g this is the way we're registering a onRequest hook today:

kibana/packages/core/http/core-http-server-internal/src/http_server.ts

Lines 491 to 500 in 8727c68

	private registerOnPreRouting(fn: OnPreRoutingHandler) {
	if (this.server === undefined) {
	throw new Error('Server is not created yet');
	}
	if (this.stopping || this.stopped) {
	this.log.warn(`registerOnPreRouting called after stop`);
	}
	this.server.ext('onRequest', adoptToHapiOnRequest(fn, this.log));
	}

Or the way we channel the server contract though our service contract:

kibana/packages/core/http/core-http-server-internal/src/http_service.ts

Lines 167 to 168 in 2aa8e6d

	this.internalSetup = {
	...serverContract,

Going down the "fully recreate the server" path would mean that we would be forced to change the way all those APIs function to make sure that the new server gets everything re-registered and that the upstream contracts are updated to use the new reference.

TLDR

• find a way to properly hot reload TLS certificates on a vanilla HAPI server

• then follow the same approach for Kibana's internal HAPI wrapper

  • use the already-in-place SIGHUP-based configuration reload
  • have the server react to those signals to recreate the TLS configuration and forward it to the HAPI server.

I'm available for any further questions.

[pgayvallet]

I opened #171823 that should close the issue (and happy thanksgiving).

[ivosh]

Thanks a lot for hot-reloading the Kibana X.509 server certificate.
Unfortunately this works only for server.ssl configuration (tested with Kibana 8.12.1).
For example certificates for elasticsearch.ssl configuration are not reloaded on SIGHUP.

For tracking:
Issue #101072 summarized Kibana settings related to X.509 certificates:

• Kibana's own HTTP server
• Kibana's connection to Elasticsearch
• Kibana's connection to the monitoring cluster (when Stack Monitoring is in use)
• Kibana's connection to the Enterprise Search service
• Kibana Alerting (TODO)

And issue #101072 has been closed as a duplicate of this issue #54368.
However #171823 fixed only the part related to server.ssl (Kibana's own HTTP server).
Reloading certificates for other purposes is not tackled, yet.

My use case is related to Kibana's connection to Elasticsearch.

[pgayvallet]

@ivosh that's fair, I reopened #101072 and updated the issue to inform only the first point (http server) was implemented.

[ivosh]

@pgayvallet Thanks for the assessment. And make no mistake, I really appreaciate your work on Kibana!