Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/transport: reload TLS certificates for every use #7784

Closed
wants to merge 1 commit into from
Closed

pkg/transport: reload TLS certificates for every use #7784

wants to merge 1 commit into from

Conversation

tgrosinger
Copy link
Contributor

This changes the baseConfig used when creating tls Configs to utilize
the GetCertificate and GetClientCertificate functions to always reload
the certificates from disk whenever they are needed.

Always reloading the certificates allows changing the certificates via
an external process without interrupting etcd.

Fixes #7576

@tgrosinger
Copy link
Contributor Author

Caveats: This requires Go 1.8.1

@gyuho
Copy link
Contributor

gyuho commented Apr 20, 2017

Commit title should be pkg/transport: reload ...?

This changes the baseConfig used when creating tls Configs to utilize
the GetCertificate and GetClientCertificate functions to always reload
the certificates from disk whenever they are needed.

Always reloading the certificates allows changing the certificates via
an external process without interrupting etcd.

Fixes #7576
@tgrosinger tgrosinger changed the title Reload TLS certificates for every use pkg/transport: reload TLS certificates for every use Apr 20, 2017
@xiang90
Copy link
Contributor

xiang90 commented Apr 20, 2017

So the existing TLS connections will not be terminated even if the cert is changed? Is this the expected behavior?

@tgrosinger
Copy link
Contributor Author

That is correct. Because the TLS negotiation occurs only when the connection is opened, ongoing connection can continue.

@xiang90
Copy link
Contributor

xiang90 commented Apr 20, 2017

@tgrosinger

yea. i know how tls works. my concerns is that users might expect the existing tls connections to be terminated when they switch key/cert. but i am not a security expert and do not know the common practice. it would be great if you can feed me any related info: like what other projects do online cert/key change. thanks.

@Spindel
Copy link

Spindel commented Apr 20, 2017

Not terminating existing connections is the usual case. Main reason to replace certs is local renewal. In the more rare case of moving from one trust root to another, it's a conscious act, and restarting the service would be a good way to terminate all connections.

Not killing existing connections matches expectation (either automatic reload, or reloading on a SIGHUP.)


// Load the certificate from disk every time so when it is replaced we
// will be using the latest version
return tlsutil.NewCert(info.CertFile, info.KeyFile, info.parseFunc)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have you benchmarked the overhead of doing this?

if this brings a huge overhead, we probably need to do caching. (inotify + caching parsed result or simply pull based caching)

ServerName: info.ServerName,
cfg.GetCertificate = func(clientHello *tls.ClientHelloInfo) (
*tls.Certificate, error) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this extra empty line at 172

}
cfg.GetClientCertificate = func(unused *tls.CertificateRequestInfo) (
*tls.Certificate, error) {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this empty line.


// Load the certificate from disk every time so when it is replaced we
// will be using the latest version
return tlsutil.NewCert(info.CertFile, info.KeyFile, info.parseFunc)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also consider caching?

@xiang90
Copy link
Contributor

xiang90 commented Apr 20, 2017

@Spindel thanks for the explanation. make sense.

@tgrosinger
Copy link
Contributor Author

For our use case we are running etcd in a Kubernetes pod, so we are using an in-memory EmptyDir volume to help with performance.

I agree that caching would be desirable, however this patch was meant to be as minimal as possible. It's something I wanted to share with the community for others to build off of, not necessarily to be merged into etcd proper as it currently stands.

@gyuho
Copy link
Contributor

gyuho commented Apr 26, 2017

Ok, I can confirm that this works with

  1. good cert etcd cluster
  2. now has bad etcd cluster
  3. client connection to this bad cert etcd cluster fails with "transport: x509: certificate has expired or is not yet valid"; please retry

Test fails because we still need Certificates: []tls.Certificate{*tlsCert} for initial listener start.
We need integration test around this, anyway. So I will cherry-pick this patch and create a separate PR with tests (today or tomorrow).

Thanks!

gyuho pushed a commit to gyuho/etcd that referenced this pull request Apr 27, 2017
This changes the baseConfig used when creating tls Configs to utilize
the GetCertificate and GetClientCertificate functions to always reload
the certificates from disk whenever they are needed.

Always reloading the certificates allows changing the certificates via
an external process without interrupting etcd.

Fixes etcd-io#7576

Cherry-picked by Gyu-Ho Lee <[email protected]>
Original commit can be found at etcd-io#7784
gyuho pushed a commit to gyuho/etcd that referenced this pull request Apr 27, 2017
This changes the baseConfig used when creating tls Configs to utilize
the GetCertificate and GetClientCertificate functions to always reload
the certificates from disk whenever they are needed.

Always reloading the certificates allows changing the certificates via
an external process without interrupting etcd.

Fixes etcd-io#7576

Cherry-picked by Gyu-Ho Lee <[email protected]>
Original commit can be found at etcd-io#7784
gyuho pushed a commit to gyuho/etcd that referenced this pull request Apr 27, 2017
This changes the baseConfig used when creating tls Configs to utilize
the GetCertificate and GetClientCertificate functions to always reload
the certificates from disk whenever they are needed.

Always reloading the certificates allows changing the certificates via
an external process without interrupting etcd.

Fixes etcd-io#7576

Cherry-picked by Gyu-Ho Lee <[email protected]>
Original commit can be found at etcd-io#7784
@gyuho
Copy link
Contributor

gyuho commented Apr 27, 2017

Closing in favor of #7829.

@gyuho gyuho closed this Apr 27, 2017
gyuho pushed a commit to gyuho/etcd that referenced this pull request Apr 27, 2017
This changes the baseConfig used when creating tls Configs to utilize
the GetCertificate and GetClientCertificate functions to always reload
the certificates from disk whenever they are needed.

Always reloading the certificates allows changing the certificates via
an external process without interrupting etcd.

Fixes etcd-io#7576

Cherry-picked by Gyu-Ho Lee <[email protected]>
Original commit can be found at etcd-io#7784
gyuho pushed a commit to gyuho/etcd that referenced this pull request Apr 27, 2017
This changes the baseConfig used when creating tls Configs to utilize
the GetCertificate and GetClientCertificate functions to always reload
the certificates from disk whenever they are needed.

Always reloading the certificates allows changing the certificates via
an external process without interrupting etcd.

Fixes etcd-io#7576

Cherry-picked by Gyu-Ho Lee <[email protected]>
Original commit can be found at etcd-io#7784
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants