etcdserver: config flag to skip verification of peer client address #10524

MartinWeindel · 2019-03-06T10:25:10Z

On initialising the peer-to-peer communication, ETCD checks the CN/SAN for the client side. This is not feasible, if the client IP address (of the HTTPS connection) is not stable or unknown at the time of creation of the certificate.

For example, if the peers of a distributed ETCD cluster with TLS enabled run in multiple clouds (or behind NAT with some kind of DHCP),
the IP addresses of the peers are not stable. Another example for such an environment is a Kubernetes cluster using Calico CNI. Here incoming requests from outside the cluster get the client address of the accepting K8s node.

Therefore a new command line option --peer-skip-client-verify is introduced to disable the client address check optionally. Basically it should be enough to rely on the knowledge of a valid certificate.

This partially fixes #8912 "Make it possible to ignore CN/SAN mismatches" (if certificates are created for DNS names and DNS records are updated according to the current proxy IP addresses).

codecov-io · 2019-03-06T13:22:50Z

Codecov Report

Merging #10524 into master will increase coverage by 0.13%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #10524      +/-   ##
==========================================
+ Coverage   63.64%   63.77%   +0.13%     
==========================================
  Files         401      401              
  Lines       37470    37473       +3     
==========================================
+ Hits        23846    23899      +53     
+ Misses      12016    11938      -78     
- Partials     1608     1636      +28

Impacted Files	Coverage Δ
etcdmain/config.go	`83.03% <100%> (+0.07%)`	⬆️
pkg/transport/listener.go	`51.35% <100%> (+1.8%)`	⬆️
client/keys.go	`55.77% <0%> (-35.68%)`	⬇️
auth/simple_token.go	`65.85% <0%> (-21.14%)`	⬇️
client/members.go	`65.32% <0%> (-20.17%)`	⬇️
clientv3/naming/grpc.go	`68.42% <0%> (-7.02%)`	⬇️
clientv3/leasing/util.go	`91.66% <0%> (-6.67%)`	⬇️
pkg/testutil/recorder.go	`77.77% <0%> (-3.71%)`	⬇️
clientv3/leasing/cache.go	`87.77% <0%> (-3.34%)`	⬇️
clientv3/balancer/balancer.go	`86.36% <0%> (-2.28%)`	⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 800e723...718e9c1. Read the comment docs.

jingyih · 2019-03-11T08:21:54Z

cc @gyuho

jingyih · 2019-03-11T18:10:16Z

cc @wenjiaswe

afritzler · 2019-04-01T11:16:28Z

Any chance this gets merged anytime soon?

MartinWeindel · 2019-04-23T12:41:02Z

Ping. Any chances for this mini pull request consisting of just 6 new lines?

wenjiaswe · 2019-04-23T23:34:14Z

@MartinWeindel Thanks for the PR! Sorry for the delay, would you please add a test to it? Otherwise lgtm

@hexfusion would you please also take a look to make sure?

gyuho · 2019-04-24T05:53:33Z

kubernetes/kubernetes#21283 (comment)

I would say no. It's not an option supported by the go http transport, and trusting the signature but not verifying the subject/sans means you are extending your trust beyond the CA to everyone holding a certificate ever issued by the CA.

/cc @liggitt

MartinWeindel · 2019-04-24T06:46:26Z

That’s exactly the purpose of this PR: to trust the certificates issued by the *private* CA. As I tried to explain is it not possible to check the subject/sans on the client side if the ETCD server instances are living in different networks with NAT. And just to repeat it. This is only an advanced option for non-standard scenarios, which has to be activated explicitly. The only possible alternative I can think of in our scenario is to set up a VPN, which is quite an effort, if you only need it for the internal communication of a distributed ETCD cluster.

MartinWeindel · 2019-04-29T06:54:05Z

@wenjiaswe As there was no test coverage for checking the client address from its certificate, the added test checks the correct behaviour for three cases:

Client IP address matches the SAN entry of the client certificate and handshaking is successful.
Client IP address does not match and handshaking fails.
SkipClientVerify flag is set and handshaking succeeds even if client IP address does not match.

wenjiaswe · 2019-05-13T17:03:27Z

Friendly ping @gyuho @liggitt for your opinion, cc @hexfusion as well. Thanks!

pkg/transport/listener_test.go

gyuho · 2019-06-10T20:47:33Z

etcdmain/config.go

@@ -210,6 +210,7 @@ func newConfig() *config {
 	fs.StringVar(&cfg.ec.PeerTLSInfo.CRLFile, "peer-crl-file", "", "Path to the peer certificate revocation list file.")
 	fs.StringVar(&cfg.ec.PeerTLSInfo.AllowedCN, "peer-cert-allowed-cn", "", "Allowed CN for inter peer authentication.")
 	fs.Var(flags.NewStringsValue(""), "cipher-suites", "Comma-separated list of supported TLS cipher suites between client/server and peers (empty will be auto-populated by Go).")
+	fs.BoolVar(&cfg.ec.PeerTLSInfo.SkipClientVerify, "peer-skip-client-verify", false, "Skip client IP verification for peer connections.")


Skip client SAN field verification for peer connections?

I think the skip client verify is a very vague description. can we make it clear to include what exactly is skipped?

Changed description to Skip verification of SAN field in client certificate for peer connections.
Any more ideas?

Skip verification of SAN field in client certificate for peer connections

This is clearer.

gyuho · 2019-06-10T20:49:13Z

I will check with SIG-auth https://github.com/kubernetes/community/tree/master/sig-auth. Overall, the use case is reasonable.

gyuho · 2019-06-11T17:47:22Z

@MartinWeindel can you update commit title to *: ...?

gyuho · 2019-06-11T17:47:55Z

/cc @deads2k @liggitt @mikedanese

xiang90 · 2019-06-11T21:31:18Z

etcdmain/config.go

@@ -210,6 +210,7 @@ func newConfig() *config {
 	fs.StringVar(&cfg.ec.PeerTLSInfo.CRLFile, "peer-crl-file", "", "Path to the peer certificate revocation list file.")
 	fs.StringVar(&cfg.ec.PeerTLSInfo.AllowedCN, "peer-cert-allowed-cn", "", "Allowed CN for inter peer authentication.")
 	fs.Var(flags.NewStringsValue(""), "cipher-suites", "Comma-separated list of supported TLS cipher suites between client/server and peers (empty will be auto-populated by Go).")
+	fs.BoolVar(&cfg.ec.PeerTLSInfo.SkipClientVerify, "peer-skip-client-verify", false, "Skip verification of SAN field in client certificate for peer connections.")


@MartinWeindel can we also change the flag naming to peer-skip-client-san-veirfy? not a fan of long flag, but we need to be clear on what is skipped on flag naming too.

Changed flag to peer-skip-client-san-verification

mikedanese · 2019-06-11T23:01:12Z

Can ETCD not be configured with different certs for client and server TLS auth? It seems like checkSAN should be OR'd with the AllowedCN check. This is allowed by the spec for both client and server auth, but is especially useful for client auth.

hexfusion · 2019-06-11T23:45:39Z

I would like some time to test this before we merge. In general this feels like a flag that will promote poor usage and thus reduce overall security. While I understand the reporters networking challenge I still am not convinced this is the only path forward.

xiang90 · 2019-06-14T00:24:22Z

defer to @hexfusion

MartinWeindel · 2019-06-26T12:11:54Z

@hexfusion Have you already found some time to test this?

hexfusion · 2019-07-11T17:56:09Z

@MartinWeindel we are going to talk about this today at the community meeting

hexfusion · 2019-07-11T23:29:03Z

@MartinWeindel it would be nice if we had a solid reproducible use case to test alternative solutions against. Your description has general networking detail but not enough to validate the need for this change. Can you provide a test ENV or steps to reproduce the ENV which we could clearly see the failure case? Thanks!

MartinWeindel · 2019-07-15T10:09:32Z

@hexfusion You can find a test ENV using Kubernetes services in the GitHub project MartinWeindel/etcd-peer-skip-client-verify. Feel free to contact me if anything is unclear.

hexfusion · 2019-07-29T09:48:20Z

@MartinWeindel after talking with the other maintainers we have decided to move this forward for v3.4 as an experimental flag. Can you please append experimental to your flag and update docs[1] as well as add details to changelog[2]?

[1] https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/configuration.md#experimental-flags
[2] https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.4.md

MartinWeindel · 2019-07-30T06:38:13Z

@hexfusion Thanks for your endurance

…fication

…tion

hexfusion

/lgtm

@MartinWeindel thank you for your patience.

andyliuliming · 2019-10-02T15:49:51Z

Hi @xiang90 is it possible we cherry-pick this change to the etcd 3.2/3.3 too?
if yes, I will prepare the PR, thanks :)

MartinWeindel force-pushed the peer-skip-client-verify branch 2 times, most recently from 21a6d36 to ec4196b Compare March 6, 2019 12:04

gyuho added the WIP label Apr 24, 2019

MartinWeindel force-pushed the peer-skip-client-verify branch 4 times, most recently from bd2cd6e to 296183f Compare April 28, 2019 16:53

wenjiaswe mentioned this pull request May 15, 2019

pkg/transport: Add test for "deny incoming peer certs with wrong IP SAN" #10729

Closed

gyuho reviewed Jun 10, 2019

View reviewed changes

pkg/transport/listener_test.go Show resolved Hide resolved

gyuho reviewed Jun 10, 2019

View reviewed changes

MartinWeindel force-pushed the peer-skip-client-verify branch from 7694a69 to 6a28ae2 Compare June 11, 2019 21:00

xiang90 reviewed Jun 11, 2019

View reviewed changes

MartinWeindel force-pushed the peer-skip-client-verify branch 3 times, most recently from f3ef5bc to aeaec51 Compare June 12, 2019 13:50

xiang90 assigned hexfusion Jun 14, 2019

This was referenced Jul 26, 2019

Release etcd 3.4 #10943

Closed

Make peer-to-peer TLS communication easier #10948

Closed

MartinWeindel force-pushed the peer-skip-client-verify branch from aeaec51 to ff3ded7 Compare July 30, 2019 06:26

MartinWeindel force-pushed the peer-skip-client-verify branch from ff3ded7 to 865beb4 Compare July 30, 2019 06:43

MartinWeindel added 4 commits July 30, 2019 14:01

etcdserver: Added configuration flag --peer-skip-client-verify=true

ff008d9

pkg/transport: Added test for SkipClientVerify flag.

80b4d4d

pkg/transport: Improved description of flag peer-skip-client-san-veri…

3bf4be4

…fication

etcdserver: mark flag as experimental, add to changelog and configura…

718e9c1

…tion

MartinWeindel force-pushed the peer-skip-client-verify branch from 865beb4 to 718e9c1 Compare July 30, 2019 12:02

hexfusion removed the WIP label Jul 30, 2019

hexfusion added this to the etcd-v3.4 milestone Jul 30, 2019

hexfusion approved these changes Jul 30, 2019

View reviewed changes

hexfusion merged commit 149e5dc into etcd-io:master Jul 30, 2019

jingyih mentioned this pull request Aug 16, 2019

etcdmain: update help message #11045

Merged

This was referenced Oct 8, 2019

etcdserver: cherry-pick skip client san verification option for 3.3 version. #11196

Merged

etcdserver: cherry-pick skip client san verification option for 3.2 version. #11195

Merged

wenjiaswe added the area/clientv3 label Jul 22, 2020

lavacat mentioned this pull request Mar 23, 2022

Track stabilization of peer-skip-client-san-verification #13778

Open

MartinWeindel deleted the peer-skip-client-verify branch March 24, 2022 07:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etcdserver: config flag to skip verification of peer client address #10524

etcdserver: config flag to skip verification of peer client address #10524

MartinWeindel commented Mar 6, 2019

codecov-io commented Mar 6, 2019 •

edited

Loading

jingyih commented Mar 11, 2019

jingyih commented Mar 11, 2019

afritzler commented Apr 1, 2019

MartinWeindel commented Apr 23, 2019

wenjiaswe commented Apr 23, 2019 •

edited

Loading

gyuho commented Apr 24, 2019

MartinWeindel commented Apr 24, 2019 via email •

edited

Loading

MartinWeindel commented Apr 29, 2019

wenjiaswe commented May 13, 2019

gyuho Jun 10, 2019

xiang90 Jun 10, 2019

MartinWeindel Jun 11, 2019

gyuho Jun 11, 2019

gyuho commented Jun 10, 2019

gyuho commented Jun 11, 2019

gyuho commented Jun 11, 2019

xiang90 Jun 11, 2019

MartinWeindel Jun 12, 2019

mikedanese commented Jun 11, 2019

hexfusion commented Jun 11, 2019

xiang90 commented Jun 14, 2019

MartinWeindel commented Jun 26, 2019

hexfusion commented Jul 11, 2019

hexfusion commented Jul 11, 2019

MartinWeindel commented Jul 15, 2019

hexfusion commented Jul 29, 2019

MartinWeindel commented Jul 30, 2019

hexfusion left a comment

andyliuliming commented Oct 2, 2019 •

edited

Loading

etcdserver: config flag to skip verification of peer client address #10524

etcdserver: config flag to skip verification of peer client address #10524

Conversation

MartinWeindel commented Mar 6, 2019

codecov-io commented Mar 6, 2019 • edited Loading

Codecov Report

jingyih commented Mar 11, 2019

jingyih commented Mar 11, 2019

afritzler commented Apr 1, 2019

MartinWeindel commented Apr 23, 2019

wenjiaswe commented Apr 23, 2019 • edited Loading

gyuho commented Apr 24, 2019

MartinWeindel commented Apr 24, 2019 via email • edited Loading

MartinWeindel commented Apr 29, 2019

wenjiaswe commented May 13, 2019

gyuho Jun 10, 2019

Choose a reason for hiding this comment

xiang90 Jun 10, 2019

Choose a reason for hiding this comment

MartinWeindel Jun 11, 2019

Choose a reason for hiding this comment

gyuho Jun 11, 2019

Choose a reason for hiding this comment

gyuho commented Jun 10, 2019

gyuho commented Jun 11, 2019

gyuho commented Jun 11, 2019

xiang90 Jun 11, 2019

Choose a reason for hiding this comment

MartinWeindel Jun 12, 2019

Choose a reason for hiding this comment

mikedanese commented Jun 11, 2019

hexfusion commented Jun 11, 2019

xiang90 commented Jun 14, 2019

MartinWeindel commented Jun 26, 2019

hexfusion commented Jul 11, 2019

hexfusion commented Jul 11, 2019

MartinWeindel commented Jul 15, 2019

hexfusion commented Jul 29, 2019

MartinWeindel commented Jul 30, 2019

hexfusion left a comment

Choose a reason for hiding this comment

andyliuliming commented Oct 2, 2019 • edited Loading

codecov-io commented Mar 6, 2019 •

edited

Loading

wenjiaswe commented Apr 23, 2019 •

edited

Loading

MartinWeindel commented Apr 24, 2019 via email •

edited

Loading

andyliuliming commented Oct 2, 2019 •

edited

Loading