Reload certificate files of https.createServer() without restarting node server

[nolimitdev]

Hi, letsencrypt certificate files expires each 3 months. Is there any way to refresh certificate files without restarting node server? Because using stale/expired certificate causes error ERR_INSECURE_RESPONSE in browser.

var fs = require('fs');
var https = require('https');
var ws = require('ws').Server;
var config = require('config.js');
var certificate = {
    key: fs.readFileSync(config.sslKeyPath),
    cert: fs.readFileSync(config.sslCrtPath),
}
var httpsServer = https.createServer(certificate).listen(config.port),
var wssServer = new ws({ server : httpsServer });

// I would like to reload certificate monthly...

// solution A): just update certificate.cer since variable certificate is passed to createServer() as reference because it is Object (not primitive value)
setInterval(function() { certificate.cert = fs.readFileSync(config.sslCrtPath); console.log("reload cerfificate A"); }, 1000 * 60 * 60 * 24 * 30);
// ... no success

// solution B): update directly httpsServer.cert (yes, this property exists when you console.log(httpsServer))
setInterval(function() { httpsServer.cert = fs.readFileSync(config.sslCrtPath); console.log("reload cerfificate B"); }, 1000 * 60 * 60 * 24 * 30);
// ... property is updated but no success

No solution works and node always use stale certificate for new incoming https requests and websocket connections too . It would be great to have a new method in returned Object from https.createServer() to reload certificate files e.g.:
httpsServer.reloadCertificate({key: fs.readFileSync(config.sslKeyPath), cert: fs.readFileSync(config.sslCrtPath)})
... now, new incoming https requests or websocket connections should be handled with new certificate files

[benjamingr]

That's interesting.

In practice, people solve this by running Node.js servers in a cluster of several instances, and then do a one-by-one update where severs individually go down and up with the new certificate so there is no runtime.

WebSocket requests require "draining", so you stop routing connections to the server and then restart it once all the existing requests are done.

I can see the use case here.

[mscdex]

FWIW you can already do this with SNICallback():

const https = require('https');
const tls = require('tls');
const fs = require('fs');
var ctx = tls.createSecureContext({
  key: fs.readFileSync(config.sslKeyPath),
  cert: fs.readFileSync(config.sslCrtPath)
});
https.createServer({
  SNICallback: (servername, cb) => {
    // here you can even change up the `SecureContext`
    // based on `servername` if you want
    cb(null, ctx);
  }
});

With that, all you have to do is re-assign ctx and then it will get used for any future requests.

[nolimitdev]

@mscdex thank you for great working fallback (already implemented) :)
So clients without SNI support will not connect now, yes? I know, it must be very old device which do not support SNI. Also option SNICallback seems to mean a little overhead or?

[mscdex]

@nolimitdev You should be able to still supply key and cert in the createServer() options in addition to SNICallback(). That pair should be used for non-SNI clients IIRC (this should be rare in this day and age though).

As far as overhead goes, it's just a couple of extra function calls really, I wouldn't be so worried about it.

[silverwind]

I wouldn't rule out non-SNI completely. There are still clients that don't support it or handle it inproperly, like the OpenSSL version included on the latest macOS ref.

I think adding a method to update server options at runtime would be nice to have. Similar to what is possible for ticket keys using setTicketKeys.

[bnoordhuis]

Duplicate of #4464?

[silverwind]

Yes, seems like a dupe.

[oxygen]

SNICallback is only invoked once when the https server is created.
Even running .close() and then .listen() won't determine SNICallback to be called again for the same domain.

As a result, SNICallback cannot be used to reload certificates.

I couldn't find any API supported way of reloading certificates.

[bnoordhuis]

SNICallback is only invoked once when the https server is created.

I'm not sure where you get that from. The SNICallback is inherited by incoming connections from the tls.Server instance on which they arrive:

node/lib/_tls_wrap.js

Lines 1054 to 1076 in 138eb32

	function tlsConnectionListener(rawSocket) {
	debug('net.Server.on(connection): new TLSSocket');
	const socket = new TLSSocket(rawSocket, {
	secureContext: this._sharedCreds,
	isServer: true,
	server: this,
	requestCert: this.requestCert,
	rejectUnauthorized: this.rejectUnauthorized,
	handshakeTimeout: this[kHandshakeTimeout],
	ALPNProtocols: this.ALPNProtocols,
	SNICallback: this[kSNICallback] || SNICallback,
	enableTrace: this[kEnableTrace],
	pauseOnConnect: this.pauseOnConnect,
	pskCallback: this[kPskCallback],
	pskIdentityHint: this[kPskIdentityHint],
	});
	socket.on('secure', onServerSocketSecure);
	socket[kErrorEmitted] = false;
	socket.on('close', onSocketClose);
	socket.on('_tlsError', onSocketTLSError);
	}

https.Server inherits that behavior from tls.Server.

[oxygen]

I tried using it.
SNICallback is called only once per domain on the https server, that is, on the first connection with a particular domain.

Subsequent connections on the same domain do not go to SNICallback and so are using some kind of cache.

NodeJS 11.7

[bnoordhuis]

It sounds like you're getting tripped up by TLS session resumption, either server-side (session IDs) or client-side (ticket keys.)

It cuts short the TLS handshake but it means SNICallback won't be invoked because there's no need, that part of the handshake is skipped.

You can disable it (at the cost of reduced handshake performance) but you don't have to. Sessions resume from the key exchange step of the handshake, they don't look (and don't have to look) at the certificate. In other words, it's immaterial that SNICallback isn't called.

https://hpbn.co/transport-layer-security-tls/#tls-session-resumption is the best overview I could find in 30 seconds of googling, hope that helps. :-)

[oxygen]

Thanks for taking the time to explain it to me. I’ll try with multiple clients and see how it goes.

[masx200]

#34444

[GoJermangoGo]

I was finally able to get this working without a full restart using tls.Server.setSecureContext. WebSocket connections using the old secure context remain open, while new WebSocket connections will use the fresh certificate.

Time will tell if it works specifically with certbot, but I'm able to trigger a reload by overwriting the certificates with self-signed garbage (obviously back up the old ones first if you want to test). cat fullchainbad.pem > fullchain1.pem && cat privkeybad.pem > privkey1.pem

Oct 2023 update: See my followup below for code example

[nolimitdev]

@luisfonsivevo I think your implementation is unclear and complicated. Try to check this #4464 (comment) I do not use setSecureContext() because it was added in node v11 and my post is from january 2018 but you can use algorithm. I just used setTimeout()+clearTimeout() without needing sth. like your "reloading". You uselessly combined setTimeout() and "reloading". I also wrote there that it is useless to watch both private key and cert because when private key is changed also certificate must be changed, so vars "keyMod" and "certMod" are also useless. Maybe you should also use chokidar instead of native watch which is platform unreliable (I fixed possible problems by clearTimeout()). I use my algorithm for several years without problems.

[GoJermangoGo]

I think that hoping both certificates are finished writing to disk after 5 seconds would be unreliable. It's pretty unlikely to ever take longer, but still. I could definitely lose an fs.watch though.

[mohsenasm]

So, to reload the certificate, can we use something like this code now?

const wsServer = new ws.Server({ server: httpsServer });
httpsServer.listen(...);

setInterval(function() { 
    const privateKey = fs.readFileSync(keyPath, 'utf8');
    const certificate = fs.readFileSync(crtPath, 'utf8');
    const credentials = { key: privateKey, cert: certificate };
    httpsServer.setSecureContext(credentials)
}, 1000 * 60 * 60 * 24);

[GoJermangoGo]

I've been using this for years now with no issue:

wsServer.reloadCerts = async function(firstLoad)
{
	while(true)
	{
		try
		{
			//will throw error if certs haven't finised writing to disk
			httpServer.setSecureContext(
			{
				key: fs.readFileSync(o.privkey),
				cert: fs.readFileSync(o.fullchain)
			});
			
			wsServer.log("Certificates reloaded");
			break;
		}
		catch(oops)
		{
			console.error(oops);
			
			if(firstLoad)
				process.exit(1);
		}
		
		await new Promise((resolve) => setTimeout(resolve, 100));
	}
};

wsServer.reloadCerts(true); //initial load
certInterval = setInterval(() => wsServer.reloadCerts(false), 24 * 60 * 60 * 1000); //daily reload

And here's a helper bash script to use it with certbot+letsencrypt (needs to run each time node.js is updated):

#helper script if you are using certbot

#allow node to open ports lower than 1024 (specifically 443 for https/wss)
sudo setcap 'cap_net_bind_service=+ep' `which node`

#allow your user (and node) to read the cert files so that node.js doesn't need to run as root
sudo addgroup nodecert
sudo adduser $USER nodecert
sudo adduser root nodecert
sudo chgrp -R nodecert /etc/letsencrypt/live
sudo chgrp -R nodecert /etc/letsencrypt/archive
sudo chmod -R 750 /etc/letsencrypt/live
sudo chmod -R 750 /etc/letsencrypt/archive

echo "If this is the first time you've run this script on this system, please REBOOT."

[nolimitdev]

Im using much easier code base on watching file. @luisfonsivevo see interval in watchFile which solves your problem will throw error if certs haven't finished writing to disk.

For compatibility with old nodejs versions:

fs.watchFile(config.tlsCer, { interval : 100 }, () => {
    console.log('Reloading TLS certificate');
    server._sharedCreds.context.setCert(fs.readFileSync(config.tlsCer));
    server._sharedCreds.context.setKey(fs.readFileSync(config.tlsKey));
});

Nowadays setSecureContext can be used:

fs.watchFile(config.tlsCer, { interval : 100 }, () => {
    console.log('Reloading TLS certificate');
    server.setSecureContext({
        cert : fs.readFileSync(config.tlsCer),
        key : fs.readFileSync(config.tlsKey),
    });
});