Error with secured etcd

[henriavelabarbe]

Hi,
after having successfully tested etcd, patroni and vip-manager on a 3 nodes cluster, I got some issue with vip-manager when etcd is secured.

etcd error: client: etcd cluster is unavailable or misconfigured; error #0: x509: certificate signed by unknown authority

Patroni is configured to use the certificates and is working well.

Could you tell me if vip-manager need some additional configuration to support certificates with etcd ?

And thank you for the tool, very usefull.

Regards,
Henri Chapelle

[martinseener]

Hi,
glad to hear that. We also use Patroni and vip-manager together with Consul instead of etcd for our 3-node Cluster.
The problem is obvious there, don't use self-signed certificates or if you bought them from a trusted CA, make sure that you also add the intermediate certificates in the cert chain in etcd and make sure that the hosts, where vip-manager runs on has a recent list of trusted CA's (so called trust store). For Linux there is a package for that, called ca-certificates.

Here you can read more about TLS Cert Chains: https://www.sysorchestra.com/ssltls-certificate-chain-challenges/

Feel free to ask here, if something is unclear.
Cheers

[henriavelabarbe]

Hi,
thanks for your answer.
I think I correctly added the CA certificate but yet it doesn't work with vip-manager.
But I'm not an expert in ssl, tls and certificates..

I'm stil wondering how vip-manager will know what key it should use to communicate with etcd ?
In patroni, we need to specify it in the configuration file AFAIK.

[martinseener]

Hi,
i still don't get it completely. Can you maybe post some configuration, so i can better understand?
Cheers

[Wolfsrudel]

I have the same problem. I've already added my ca to the trust store, but the missing part (in the documentation) is how vip-manager get to know about the certificates it should use.
Environment variables doesn't seem to work.

ETCDCTL_API=3 ETCDCTL_CACERT=/var/lib/postgresql/certs/ca.pem \ ETCDCTL_CERT=/var/lib/postgresql/certs/client.pem \ ETCDCTL_KEY=/var/lib/postgresql/certs/client-key.pem \ /usr/bin/vip-manager -ip="x.x.x.x" -iface="eth0" -key="/service/pgcluster/leader" -host="my-hostname" -type="etcd" -endpoint="https://my-hostname:2379" -mask="24"

2019/05/23 10:22:19 IP address x.x.x.x/24 state is false, desired false 2019/05/23 10:22:19 etcd error: client: etcd cluster is unavailable or misconfigured; error #0: remote error: tls: bad certificate 2019/05/23 10:22:20 etcd error: client: etcd cluster is unavailable or misconfigured; error #0: remote error: tls: bad certificate

[martinseener]

Most likely this is something the etcd client does and not vip-manager. One has to check the etc client code and check where it performs the TLS stuff. vip-manager has to be updated at some point because when i'm right, the etcd client has moved to https://github.com/etcd-io/etcd/tree/master/client. @Wolfsrudel you may want to check the code there to see how it does TLS verifying?

[Wolfsrudel]

Sorry, but I'm not a developer. :)

[Wolfsrudel]

Now I see: you are using the older client version. Maybe it is working with APIv2.

[Wolfsrudel]

Same error using
Environment="ETCDCTL_API=2"
Environment="ETCDCTL_CA_FILE=ca.pem"
Environment="ETCDCTL_CERT_FILE=client.pem"
Environment="ETCDCTL_KEY_FILE=client-key.pem"
parameters in the unit file of systemd.

[martinseener]

I'm also not a programmer, but we could go through the source code step by step. The problem also is that 'we' use consul as a backend and not etcd.

[Wolfsrudel]

I'm also not a programmer, but we could go through the source code step by step. The problem also is that 'we' use consul as a backend and not etcd.

Secured with tls? If true i'll give it a try.

[Wolfsrudel]

It's working with a secured consul. It's fine for me.

[martinseener]

@Wolfsrudel nice to hear that ;) At least you know, that i can help you when you have some consul related questions. Our PG Cluster (3-node) now works perfectly fine since November 2017 with patroni/consul/vip-manager :)

[markwort]

Hey guys, since I added options to specify etcd username and password, it should be fairly easy to extend the yaml config reader and the etcd checker (same applies to consul checker...) to use user-specified cert, key, and ca files.
Just a hint if you'd like to create a small pull request :)

I may get around to do this at some point, but the demand seems to be low currently. (Please complain loudly if you disagree.)

[gandalfmagic]

@markwort I've resolved the issue creating a client.CancelableTransport with TLS enabled in the EtcdLeaderChecker object.

I'm using this set of environment variables to enable the TLS connection:

• ETCD_CLIENT_CERT_AUTH: if set to true, we enable a TLS connection
• ETCD_TRUSTED_CA_FILE: must be set to the path of the etcd CA certificate file
• ETCD_CERT_FILE: must be set to the path of the etcd client certificate file
• ETCD_KEY_FILE: must be set to the path of the etcd client key file

I can create a PR whenever you want.

[markwort]

@gandalfmagic so you're using Environment variables to influence the behaviour of vip-manager in respect to the etcd client?
Why not flags like all other parameters, or the new yaml config? :)

If you'd like to create a pull request, I can check how we can make it work with a configuration that is consistent with the rest of the code.

[gandalfmagic]

Hi @markwort,

@gandalfmagic so you're using Environment variables to influence the behaviour of vip-manager in respect to the etcd client?

In this case I wanted to avoid a different behaviour of etcd compared to consul.

Why not flags like all other parameters, or the new yaml config? :)

Can we support both? flags/parameters and env variables? Is it fine for you?

I'm testing the new code right now, if you tell me you prefer the solution without env variables, I'll simply drop the support to them, no problem :)

[tpo]

For people having the same problem (me for example), @gandalfmagic 's code is here: https://github.com/gandalfmagic/vip-manager

@markwort - please check @gandalmagic 's repository. It implements both passing of parameters via environment and as command line options. His repository has other nice improvements too, such as gitlab-ci and test, better distribution packaging etc.

I have extracted a few "single issue" patches from @gandalmagic 's repository, that you should be able to merge easily:

• fix password parameter #27
• fix init script description #26

Mind you, if you have a look at @gandalfmagic 's work and you think you would like to pull all of it, then it's better to pull directly from him, because there's slight chance of merge conflicts if you ull the stuff I picked from @gandalfmagic 's repository first.

Note that when building from a forked repository, you (whoever is trying to build the thing) will get plenty on build errors like this:

$ make
go build -ldflags="-s -w" .
# github.com/tpo/vip-manager/checker
../../_go/src/github.com/tpo/vip-manager/checker/etcd_leader_checker.go:46:42: conf.Etcd_ca_file undefined (type vipconfig.Config has no field or method Etcd_ca_file)

because there's plenty of places where:

$ cat checker/consul_leader_checker.go
[...]
import (
    "github.com/cybertec-postgresql/vip-manager/vipconfig"
)

which need all to be changed to a repo that contains the modified vipconfig. So say you fork as my_name_on_github_is_foo then that would be:

$ cat checker/consul_leader_checker.go
[...]
import (
    "github.com/my_name_on_github_is_fooo/vip-manager/vipconfig"
)

See d58b948 for reference.

I think that's also something that @gandalfmagic has addressed by switching to a different dependency/vendoring mechanism in his repo.

[pashagolub]

Hello @gandalfmagic. I've checked your fork and found it very useful. I saw you switched to go modules, but why you still have vendor folder?

[gandalfmagic]

Hello @gandalfmagic. I've checked your fork and found it very useful. I saw you switched to go modules, but why you still have vendor folder?

Hi @pashagolub, I simply never removed it after the migration. You can do it any time.

[pashagolub]

Hello @gandalfmagic. I've checked your fork and found it very useful. I saw you switched to go modules, but why you still have vendor folder?

Hi @pashagolub, I simply never removed it after the migration. You can do it any time.

Would be great if you can drop it. It will ease merging :) Thanks in advance

[gandalfmagic]

Hello @gandalfmagic. I've checked your fork and found it very useful. I saw you switched to go modules, but why you still have vendor folder?

Hi @pashagolub, I simply never removed it after the migration. You can do it any time.

Would be great if you can drop it. It will ease merging :) Thanks in advance

Done.

[ghost]

Hi @pashagolub
I'm using the version 0.6.1

vip-manager -version
version 0.6.1

I've the same problem with certificates :

/usr/bin/vip-manager -config=/etc/patroni/vip-manager.yml
2020/09/17 11:16:21 reading config from /etc/patroni/vip-manager.yml
2020/09/17 11:16:21 IP address XX.XX.XX.XX/23 state is false, desired false
2020/09/17 11:16:21 etcd error: client: etcd cluster is unavailable or misconfigured; error #0: x509: certificate signed by unknown authority
; error #1: x509: certificate signed by unknown authority
; error #2: x509: certificate signed by unknown authority

Using etcdctl client it's working well with the same certificates :

etcdctl --cert /etc/etcd/ssl/client.crt --key /etc/etcd/ssl/client.key --cacert /etc/etcd/ssl/ca.crt endpoint health --cluster
https://XX.XX.XX.XX:2379 is healthy: successfully committed proposal: took = 16.244274ms
https://XX.XX.XX.XX:2379 is healthy: successfully committed proposal: took = 16.870579ms
https://XX.XX.XX.XX:2379 is healthy: successfully committed proposal: took = 29.936325ms

In the yml config file I added the certificates :

endpoint_type: etcd # etcd or consul
# a list that contains all endpoints to which etcd could talk.
endpoints:
  - https://127.0.0.1:2379
  - https://member2:2379
  - https://member3:2379

etcd_ca_file: "/etc/etcd/ssl/ca.crt"
etcd_cert_file: "/etc/etcd/ssl/client.crt"
etcd_key_file: "/etc/etcd/ssl/client.key"

What should I do to correct this certificates problem ?
Thanks

[pashagolub]

We didn't merge this, nor release it. Am I right @markwort?

[ghost]

ahhh my bad !
sorry I'll wait 👍

[markwort]

@pashagolub right...
@dagravito do you want to test a new version? I've done some testing already, but a second check by somebody who doesn't know how the code looks is always good :)
I can provide you with details to build the version or send you a pre-release build...

[ghost]

why not 👍
I've already tested the amd64.rpm on Redhat 7.
I don't know how to build but I can learn or if you already have a new build ready :)

[markwort]

I've created a new prerelease:
https://github.com/cybertec-postgresql/vip-manager/releases/tag/v1.0-beta2
Please make sure to check the documentation (README.md) for information about the changed configuration handling.

I'm closing this thread now, TLS connections to etcd are supported as of vip.-manager v1.0 .
Thanks to @gandalfmagic and @tpo for helping with implementing this!
@dagravito If you discover any bugs during testing, please simply create a new issue! :)

[ghost]

Hi @markwort
thanks for the build but with this one on my Redhat 7 I can't run it :)

vip-manager
vip-manager: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by vip-manager)

[markwort]

Hey, I'm sorry for the confusion, my workstation apparently is too up to date 😬 .
I've created a v1.0-beta3 release which contains packages built using a current CentOS 7 container image...
https://github.com/cybertec-postgresql/vip-manager/releases/tag/v1.0-beta3