*: fix TLS reload when cert includes only IPs (no domain names in SAN field)

CHANGELOG-3.2.md

@@ -11,7 +11,10 @@ See [code changes](https://github.com/coreos/etcd/compare/v3.2.18...v3.2.19) and
 
 ### Security, Authentication
 
-- Fix [TLS reload](TODO) when [cert SAN field only contains IP addresses](https://github.com/coreos/etcd/issues/9541).
+- Fix [TLS reload](https://github.com/coreos/etcd/pull/9570) when [certificate SAN field only includes IP addresses but no domain names](https://github.com/coreos/etcd/issues/9541).
+  - In Go, server calls `(*tls.Config).GetCertificate` for TLS reload if and only if server's `(*tls.Config).Certificates` field is not empty, or `(*tls.ClientHelloInfo).ServerName` is not empty with a valid SNI from the client. Previously, etcd always populates `(*tls.Config).Certificates` on the initial client TLS handshake, as non-empty. Thus, client was always expected to supply a matching SNI in order to pass the TLS verification and to trigger `(*tls.Config).GetCertificate` to reload TLS assets.
+  - However, a certificate whose SAN field does [not include any domain names but only IP addresses](https://github.com/coreos/etcd/issues/9541) would request `*tls.ClientHelloInfo` with an empty `ServerName` field, thus failing to trigger the TLS reload on initial TLS handshake; this becomes a problem when expired certificates need to be replaced online.
+  - Now, `(*tls.Config).Certificates` is created empty on initial TLS client handshake, first to trigger `(*tls.Config).GetCertificate`, and then to populate rest of the certificates on every new TLS connection, even when client SNI is empty (e.g. cert only includes IPs).
 
 
 ## [v3.2.18](https://github.com/coreos/etcd/releases/tag/v3.2.18) (2018-03-29)

CHANGELOG-3.3.md

@@ -11,7 +11,10 @@ See [code changes](https://github.com/coreos/etcd/compare/v3.3.3...v3.3.4) and [
 
 ### Security, Authentication
 
-- Fix [TLS reload](TODO) when [cert SAN field only contains IP addresses](https://github.com/coreos/etcd/issues/9541).
+- Fix [TLS reload](https://github.com/coreos/etcd/pull/9570) when [certificate SAN field only includes IP addresses but no domain names](https://github.com/coreos/etcd/issues/9541).
+  - In Go, server calls `(*tls.Config).GetCertificate` for TLS reload if and only if server's `(*tls.Config).Certificates` field is not empty, or `(*tls.ClientHelloInfo).ServerName` is not empty with a valid SNI from the client. Previously, etcd always populates `(*tls.Config).Certificates` on the initial client TLS handshake, as non-empty. Thus, client was always expected to supply a matching SNI in order to pass the TLS verification and to trigger `(*tls.Config).GetCertificate` to reload TLS assets.
+  - However, a certificate whose SAN field does [not include any domain names but only IP addresses](https://github.com/coreos/etcd/issues/9541) would request `*tls.ClientHelloInfo` with an empty `ServerName` field, thus failing to trigger the TLS reload on initial TLS handshake; this becomes a problem when expired certificates need to be replaced online.
+  - Now, `(*tls.Config).Certificates` is created empty on initial TLS client handshake, first to trigger `(*tls.Config).GetCertificate`, and then to populate rest of the certificates on every new TLS connection, even when client SNI is empty (e.g. cert only includes IPs).
 
 
 ## [v3.3.3](https://github.com/coreos/etcd/releases/tag/v3.3.3) (2018-03-29)

CHANGELOG-3.4.md

@@ -103,7 +103,10 @@ See [security doc](https://github.com/coreos/etcd/blob/master/Documentation/op-g
 - Support [`ttl` field for `etcd` Authentication JWT token](https://github.com/coreos/etcd/pull/8302).
   - e.g. `etcd --auth-token jwt,pub-key=<pub key path>,priv-key=<priv key path>,sign-method=<sign method>,ttl=5m`.
 - Allow empty token provider in [`etcdserver.ServerConfig.AuthToken`](https://github.com/coreos/etcd/pull/9369).
-- Fix [TLS reload](TODO) when [cert SAN field only contains IP addresses](https://github.com/coreos/etcd/issues/9541).
+- Fix [TLS reload](https://github.com/coreos/etcd/pull/9570) when [certificate SAN field only includes IP addresses but no domain names](https://github.com/coreos/etcd/issues/9541).
+  - In Go, server calls `(*tls.Config).GetCertificate` for TLS reload if and only if server's `(*tls.Config).Certificates` field is not empty, or `(*tls.ClientHelloInfo).ServerName` is not empty with a valid SNI from the client. Previously, etcd always populates `(*tls.Config).Certificates` on the initial client TLS handshake, as non-empty. Thus, client was always expected to supply a matching SNI in order to pass the TLS verification and to trigger `(*tls.Config).GetCertificate` to reload TLS assets.
+  - However, a certificate whose SAN field does [not include any domain names but only IP addresses](https://github.com/coreos/etcd/issues/9541) would request `*tls.ClientHelloInfo` with an empty `ServerName` field, thus failing to trigger the TLS reload on initial TLS handshake; this becomes a problem when expired certificates need to be replaced online.
+  - Now, `(*tls.Config).Certificates` is created empty on initial TLS client handshake, first to trigger `(*tls.Config).GetCertificate`, and then to populate rest of the certificates on every new TLS connection, even when client SNI is empty (e.g. cert only includes IPs).
 
 ### Added: `etcd`
 

Documentation/op-guide/security.md

@@ -321,6 +321,22 @@ I | embed: serving client requests on 127.0.0.1:22379
 I | embed: serving client requests on 127.0.0.1:2379
 ```
 
+[v3.2.19](https://github.com/coreos/etcd/blob/master/CHANGELOG-3.2.md) and [v3.3.4](https://github.com/coreos/etcd/blob/master/CHANGELOG-3.3.md) fixes TLS reload when [certificate SAN field only includes IP addresses but no domain names](https://github.com/coreos/etcd/issues/9541). For example, a member is set up with CSRs (with `cfssl`) as below:
+
+```json
+{
+  "CN": "etcd.local",
+  "hosts": [
+    "127.0.0.1"
+  ],
+```
+
+In Go, server calls `(*tls.Config).GetCertificate` for TLS reload if and only if server's `(*tls.Config).Certificates` field is not empty, or `(*tls.ClientHelloInfo).ServerName` is not empty with a valid SNI from the client. Previously, etcd always populates `(*tls.Config).Certificates` on the initial client TLS handshake, as non-empty. Thus, client was always expected to supply a matching SNI in order to pass the TLS verification and to trigger `(*tls.Config).GetCertificate` to reload TLS assets.
+
+However, a certificate whose SAN field does [not include any domain names but only IP addresses](https://github.com/coreos/etcd/issues/9541) would request `*tls.ClientHelloInfo` with an empty `ServerName` field, thus failing to trigger the TLS reload on initial TLS handshake; this becomes a problem when expired certificates need to be replaced online.
+
+Now, `(*tls.Config).Certificates` is created empty on initial TLS client handshake, first to trigger `(*tls.Config).GetCertificate`, and then to populate rest of the certificates on every new TLS connection, even when client SNI is empty (e.g. cert only includes IPs).
+
 ## Notes for Host Whitelist
 
 `etcd --host-whitelist` flag specifies acceptable hostnames from HTTP client requests. Client origin policy protects against ["DNS Rebinding"](https://en.wikipedia.org/wiki/DNS_rebinding) attacks to insecure etcd servers. That is, any website can simply create an authorized DNS name, and direct DNS to `"localhost"` (or any other address). Then, all HTTP endpoints of etcd server listening on `"localhost"` becomes accessible, thus vulnerable to DNS rebinding attacks. See [CVE-2018-5702](https://bugs.chromium.org/p/project-zero/issues/detail?id=1447#c2) for more detail.

integration/cluster.go

@@ -46,6 +46,7 @@ import (
 	"github.com/coreos/etcd/etcdserver/api/v3rpc"
 	pb "github.com/coreos/etcd/etcdserver/etcdserverpb"
 	"github.com/coreos/etcd/pkg/testutil"
+	"github.com/coreos/etcd/pkg/tlsutil"
 	"github.com/coreos/etcd/pkg/transport"
 	"github.com/coreos/etcd/pkg/types"
 	"github.com/coreos/etcd/rafthttp"
@@ -83,13 +84,27 @@ var (
 		ClientCertAuth: true,
 	}
 
+	testTLSInfoIP = transport.TLSInfo{
+		KeyFile:        "./fixtures/server-ip.key.insecure",
+		CertFile:       "./fixtures/server-ip.crt",
+		TrustedCAFile:  "./fixtures/ca.crt",
+		ClientCertAuth: true,
+	}
+
 	testTLSInfoExpired = transport.TLSInfo{
 		KeyFile:        "./fixtures-expired/server.key.insecure",
 		CertFile:       "./fixtures-expired/server.crt",
 		TrustedCAFile:  "./fixtures-expired/ca.crt",
 		ClientCertAuth: true,
 	}
 
+	testTLSInfoExpiredIP = transport.TLSInfo{
+		KeyFile:        "./fixtures-expired/server-ip.key.insecure",
+		CertFile:       "./fixtures-expired/server-ip.crt",
+		TrustedCAFile:  "./fixtures-expired/ca.crt",
+		ClientCertAuth: true,
+	}
+
 	plog = capnslog.NewPackageLogger("github.com/coreos/etcd", "integration")
 )
 
@@ -110,6 +125,9 @@ type ClusterConfig struct {
 
 	ClientMaxCallSendMsgSize int
 	ClientMaxCallRecvMsgSize int
+
+	// UseIP is true to use only IP for gRPC requests.
+	UseIP bool
 }
 
 type cluster struct {
@@ -248,6 +266,7 @@ func (c *cluster) mustNewMember(t *testing.T) *member {
 			grpcKeepAliveTimeout:     c.cfg.GRPCKeepAliveTimeout,
 			clientMaxCallSendMsgSize: c.cfg.ClientMaxCallSendMsgSize,
 			clientMaxCallRecvMsgSize: c.cfg.ClientMaxCallRecvMsgSize,
+			useIP: c.cfg.UseIP,
 		})
 	m.DiscoveryURL = c.cfg.DiscoveryURL
 	if c.cfg.UseGRPC {
@@ -511,6 +530,7 @@ type member struct {
 	keepDataDirTerminate     bool
 	clientMaxCallSendMsgSize int
 	clientMaxCallRecvMsgSize int
+	useIP                    bool
 }
 
 func (m *member) GRPCAddr() string { return m.grpcAddr }
@@ -527,6 +547,7 @@ type memberConfig struct {
 	grpcKeepAliveTimeout     time.Duration
 	clientMaxCallSendMsgSize int
 	clientMaxCallRecvMsgSize int
+	useIP                    bool
 }
 
 // mustNewMember return an inited member with the given name. If peerTLS is
@@ -600,6 +621,7 @@ func mustNewMember(t *testing.T, mcfg memberConfig) *member {
 	}
 	m.clientMaxCallSendMsgSize = mcfg.clientMaxCallSendMsgSize
 	m.clientMaxCallRecvMsgSize = mcfg.clientMaxCallRecvMsgSize
+	m.useIP = mcfg.useIP
 
 	m.InitialCorruptCheck = true
 
@@ -610,6 +632,9 @@ func mustNewMember(t *testing.T, mcfg memberConfig) *member {
 func (m *member) listenGRPC() error {
 	// prefix with localhost so cert has right domain
 	m.grpcAddr = "localhost:" + m.Name
+	if m.useIP { // for IP-only sTLS certs
+		m.grpcAddr = "127.0.0.1:" + m.Name
+	}
 	l, err := transport.NewUnixListener(m.grpcAddr)
 	if err != nil {
 		return fmt.Errorf("listen failed on grpc socket %s (%v)", m.grpcAddr, err)
@@ -785,10 +810,45 @@ func (m *member) Launch() error {
 		if m.ClientTLSInfo == nil {
 			hs.Start()
 		} else {
-			hs.TLS, err = m.ClientTLSInfo.ServerConfig()
+			info := m.ClientTLSInfo
+			hs.TLS, err = info.ServerConfig()
 			if err != nil {
 				return err
 			}
+
+			// baseConfig is called on initial TLS handshake start.
+			//
+			// Previously,
+			// 1. Server has non-empty (*tls.Config).Certificates on client hello
+			// 2. Server calls (*tls.Config).GetCertificate iff:
+			//    - Server's (*tls.Config).Certificates is not empty, or
+			//    - Client supplies SNI; non-empty (*tls.ClientHelloInfo).ServerName
+			//
+			// When (*tls.Config).Certificates is always populated on initial handshake,
+			// client is expected to provide a valid matching SNI to pass the TLS
+			// verification, thus trigger server (*tls.Config).GetCertificate to reload
+			// TLS assets. However, a cert whose SAN field does not include domain names
+			// but only IP addresses, has empty (*tls.ClientHelloInfo).ServerName, thus
+			// it was never able to trigger TLS reload on initial handshake; first
+			// ceritifcate object was being used, never being updated.
+			//
+			// Now, (*tls.Config).Certificates is created empty on initial TLS client
+			// handshake, in order to trigger (*tls.Config).GetCertificate and populate
+			// rest of the certificates on every new TLS connection, even when client
+			// SNI is empty (e.g. cert only includes IPs).
+			//
+			// This introduces another problem with "httptest.Server":
+			// when server initial certificates are empty, certificates
+			// are overwritten by Go's internal test certs, which have
+			// different SAN fields (e.g. example.com). To work around,
+			// re-overwrite (*tls.Config).Certificates before starting
+			// test server.
+			tlsCert, err := tlsutil.NewCert(info.CertFile, info.KeyFile, nil)
+			if err != nil {
+				return err
+			}
+			hs.TLS.Certificates = []tls.Certificate{*tlsCert}
+
 			hs.StartTLS()
 		}
 		closer := func() {

integration/fixtures-expired/ca.crt

@@ -1,23 +1,23 @@
 -----BEGIN CERTIFICATE-----
-MIID0jCCArqgAwIBAgIUbY6SSy/rF2TQzWsH4GxG+h+Pvw8wDQYJKoZIhvcNAQEL
+MIID0jCCArqgAwIBAgIUEKEIOO1O97Bz4car+7SHDxT5tB4wDQYJKoZIhvcNAQEL
 BQAwbzEMMAoGA1UEBhMDVVNBMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQH
 Ew1TYW4gRnJhbmNpc2NvMQ0wCwYDVQQKEwRldGNkMRYwFAYDVQQLEw1ldGNkIFNl
-Y3VyaXR5MQswCQYDVQQDEwJjYTAeFw0xODA0MDgxNzUzMDBaFw0yODA0MDUxNzUz
+Y3VyaXR5MQswCQYDVQQDEwJjYTAeFw0wMDA0MTMxODU1MDBaFw0xMDA0MTExODU1
 MDBaMG8xDDAKBgNVBAYTA1VTQTETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UE
 BxMNU2FuIEZyYW5jaXNjbzENMAsGA1UEChMEZXRjZDEWMBQGA1UECxMNZXRjZCBT
 ZWN1cml0eTELMAkGA1UEAxMCY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
-AoIBAQCqhEOeNSLK5CcfvZgHFHPJzRWeDc/fAQ3U2GSF1+KEslOA0mmHiL1paloS
-CbuwzoY/EGPCudFxIwFwjl2BAxbMdaCAKCxPwMHfn/38I45GgJFODjcOP0AX9i3O
-z2jsAGm02HNicmF24TuQgij8lvhhKjNsy2Lrb8/i6NmX8AKZl9smkRRd5HpUz9DD
-HelH2CXYCjbGXdpCyjN2PwfGSoCsAV8NDwbe0CAg6+dZCQrbqt2PJE2uRBoLgp3p
-AsVdPiFL1igOimgQRShGvMEVLkA7cmB3fALZy1WTGGj4h76HtEz8nywN7PmoWQJv
-AZFM168XPQ35S9+1CROtWUoM7dlhAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjAS
-BgNVHRMBAf8ECDAGAQH/AgECMB0GA1UdDgQWBBSLaEU8nqrYzNEcmi0oZKd1AAFK
-gTAfBgNVHSMEGDAWgBSLaEU8nqrYzNEcmi0oZKd1AAFKgTANBgkqhkiG9w0BAQsF
-AAOCAQEApPHGwdcMRWMk+RS1NVb3yCPdf2Tx8pPYAJpLY46OPenGnFt6+wJs6Nhq
-bj9zmEEqyn1WLXtuel+X4E4BEofkTEAM+06UT7SGgEF7zMY+zQjfPqD52jLhS11I
-hp3u/hDR5c8r6RmvuH1TiPK5twxmV1w6LRGQcGJtw1PdTVfgHM+1s7kQ+Ineo4kK
-8m1JR44B3GHyw+o0jsf5NqnmQnW6aMACQXiX93fnelkPOsKez/oxiy/WK5dDMrzH
-JgNonK+bZRpef15XK3EOhmHp8YrY0CEq4MFsxxmkMZT0OnvIMEi9SkPV1cFq2N7r
-uTB9aMzzD/1u+3+IpHCrkb0QICj3YQ==
+AoIBAQDFeNJ9r2TFcJp9UHS42QN2NN1A96LQXxn/BirHzXdeTk6YEe0eloA91SJT
+BAae7aGdPMkpMyAAXheGPGHAbSde5dONYx2QE4nqWRl79v6kDbX6EmqwpzTOGD/T
+UfLXe65g6w6kaXcNZWiMdqfkUImke/WWM1qunsCKoGOXF8Jg1DLy7NSjqT2Kg1UP
+evJ5GOrWmIj5rEnEvW0ohR7mKV23xl5okVjrlzCi+arWDdl5RzE0I9x7vKNE0TKX
+NNHG9hMSJQ/ipXXXyMcahqGZXtkGvOpwpO3lpsGjo3WIUZMQW2FA3xR0nBC6Lt+0
+d+7IXOy/LbzXpkcL8Ws5BZuLDSKLAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjAS
+BgNVHRMBAf8ECDAGAQH/AgECMB0GA1UdDgQWBBT5Z7kwwTNntO1UsuMdUv/Yq3Ad
+cDAfBgNVHSMEGDAWgBT5Z7kwwTNntO1UsuMdUv/Yq3AdcDANBgkqhkiG9w0BAQsF
+AAOCAQEAkFF/fIVlcgj1uRL36wP4DgkOMCpU5+vwlDdHihDzRJHZqik+3+oNz7DD
+pRIURHMeeF+Wk5/GRQ/oGzKYotNLLzqCOggnLCxET6Hkb07vfve91HmYVOYix5pU
+GPW8+M3XyFTL3+2BnPpqPpJWpJ28g+N3eQjAG8rIbjXESdxrpJFKY22nMbtyS1rH
+dyzf3OO4S7LZiRQx0nuD9SZtX2vj5DyN8Am/zieSYm+GCtJsvIiDoB+Uhndnxxt0
+FA0/89vGJ1gCo+Z6clzqBIbesRUBnLvPbUdpxhFAtjUKZhQv05IrE81/GP7F7kEr
+oODS2+D5WC6mKDO4v2k736OTw6HwOQ==
 -----END CERTIFICATE-----

integration/fixtures-expired/gencerts.sh

@@ -24,8 +24,19 @@ cfssl gencert \
   ./server-ca-csr.json | cfssljson --bare ./server
 mv server.pem server.crt
 mv server-key.pem server.key.insecure
+
+# generate IP: 127.0.0.1, CN: example.com certificates
+cfssl gencert \
+  --ca ./ca.crt \
+  --ca-key ./ca-key.pem \
+  --config ./gencert.json \
+  ./server-ca-csr-ip.json | cfssljson --bare ./server-ip
+mv server-ip.pem server-ip.crt
+mv server-ip-key.pem server-ip.key.insecure
+
 if which openssl >/dev/null; then
   openssl x509 -in ./server.crt -text -noout
+  openssl x509 -in ./server-ip.crt -text -noout
 fi
 
 rm -f *.csr *.pem *.stderr *.txt

integration/fixtures-expired/server-ca-csr-ip.json

@@ -0,0 +1,19 @@
+{
+  "key": {
+    "algo": "rsa",
+    "size": 2048
+  },
+  "names": [
+    {
+      "O": "etcd",
+      "OU": "etcd Security",
+      "L": "San Francisco",
+      "ST": "California",
+      "C": "USA"
+    }
+  ],
+  "CN": "example.com",
+  "hosts": [
+    "127.0.0.1"
+  ]
+}

integration/fixtures-expired/server-ip.crt

@@ -0,0 +1,24 @@
+-----BEGIN CERTIFICATE-----
+MIIEBzCCAu+gAwIBAgIUOc4vrxQ6OeHoGslhL7daP1Ye8ZYwDQYJKoZIhvcNAQEL
+BQAwbzEMMAoGA1UEBhMDVVNBMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQH
+Ew1TYW4gRnJhbmNpc2NvMQ0wCwYDVQQKEwRldGNkMRYwFAYDVQQLEw1ldGNkIFNl
+Y3VyaXR5MQswCQYDVQQDEwJjYTAeFw0wMDA0MTMxODU1MDBaFw0wMDA0MTMxOTU1
+MDBaMHgxDDAKBgNVBAYTA1VTQTETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UE
+BxMNU2FuIEZyYW5jaXNjbzENMAsGA1UEChMEZXRjZDEWMBQGA1UECxMNZXRjZCBT
+ZWN1cml0eTEUMBIGA1UEAxMLZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUA
+A4IBDwAwggEKAoIBAQDLUZuwXwiky2VvujM0EOP9LL85sx1dKLc/16hOl6qYRPOg
+PH7zsmXMVndsD2Fi9NDbhV9rVHfVNkCyZO/D81u52UEyr2uSFHOfIqLkFGKvcxhO
+FtOLA7wTjzHiYO1pFgqYBWzSfIyreYYo13tCYxHUlhn3ibqvCz9fimGsQmswhUiP
+yaC4C8iBICWNd4vrXHhtKb5pHHzUDFHkOxKF6VS9f7InKBy2yTr8ekgoEYyE3gtp
+ncoVbVlwxehChbZThFi0xsQc/kG/eoyGznKo9RUlUW+h3SEJR3bYizYP76ZwWXus
+nP5vgLmZ0wIi/689uTQbEAK438rK3xTSziPv6B51AgMBAAGjgZEwgY4wDgYDVR0P
+AQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMB
+Af8EAjAAMB0GA1UdDgQWBBR8bgC5SVrIrOAkjED8FB6OThk0IjAfBgNVHSMEGDAW
+gBT5Z7kwwTNntO1UsuMdUv/Yq3AdcDAPBgNVHREECDAGhwR/AAABMA0GCSqGSIb3
+DQEBCwUAA4IBAQDEV4Ec8TpDXvFTJYXrpME5KnKvtq1fEv100jc88cmlGb/rygge
+MtisA1rYSaSEPMF0j7HoMtTwP90yrJCBTr7/vziAXCZU2H6bg24exRzqtMDpDhXg
+mvqkqvMVFem8ANIF3a+qXPY/pzjh4xrPuOw10TfG0bE576lAY/KbnY3UvXo6QL54
+AMyimFhq8e9dJ7JnO3eaYmJv6oSjKjqNYSU+01UfxEJGNbx1IELMDlnVKX0Zmn9p
+YbUS3nrowKoVXpuca9KzS1pINgqVsztF5XJxzqlcDwERR/QcTKwUgQ0y0BBRqiGg
+WdtbyamFufvF8GPsNJ0KRHXSIRRXF7hbgiXd
+-----END CERTIFICATE-----

integration/fixtures-expired/server-ip.key.insecure

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEogIBAAKCAQEAy1GbsF8IpMtlb7ozNBDj/Sy/ObMdXSi3P9eoTpeqmETzoDx+
+87JlzFZ3bA9hYvTQ24Vfa1R31TZAsmTvw/NbudlBMq9rkhRznyKi5BRir3MYThbT
+iwO8E48x4mDtaRYKmAVs0nyMq3mGKNd7QmMR1JYZ94m6rws/X4phrEJrMIVIj8mg
+uAvIgSAljXeL61x4bSm+aRx81AxR5DsShelUvX+yJygctsk6/HpIKBGMhN4LaZ3K
+FW1ZcMXoQoW2U4RYtMbEHP5Bv3qMhs5yqPUVJVFvod0hCUd22Is2D++mcFl7rJz+
+b4C5mdMCIv+vPbk0GxACuN/Kyt8U0s4j7+gedQIDAQABAoIBAAjHl1+AWxEyr0ip
+07g12oJ+QiutrmDtdyxMlbn/FqDIqXSL6DeBxp+SREnoSB5L0BEKq1opJZuRYi3R
+6gCeK6HU3dngdVazh2KhzkLnFnPZFn2Ywr3IBYEat968rMPS7dYutcpJEpH9B2wQ
+EgSF3qk9ahWkXulcJPptMVaM77ACnZk6yYsPDPqjX/zsCXVga59QL0x1n2ai50er
+W7kthCj69zZP6crbnjyCUDjNdpDio7xurvvxs0k1KWmcN9QjdOyFXDFgTUnphIFX
+pEyVhu+LmLzFKc1WVQ4sIAt6ot9kpWt+cdaBVIWl2yCmqF4nbJ38DDG6wLXaZQd1
+DgEL0YECgYEAzt7QukPfjgw5CKZguQVVn0LGYdHw47qHiusjABzYH4mokMHqR/r5
+LIIRQ4JjB/vpxavj6B0e73tcfwbSzLSwsRI9/6Z27UVpXnpU5LY7+46d+ZXsQorE
+8jeUX6ZQi65ujpFFKkftKlmq67XJtmSh2T+3dMqRXmFWVZThllBJcGUCgYEA+5rd
+gvZhaj9Rng1CwK3FoI/mp0BtSL+TE8/JbV0yA5X6NhXlts/ysafFZsj9RkR1NhXL
+ql8Bl9RxrV6mTIz6/76NC39ZUQUe5FZGv64rqjoFwOnv6ap1/8ntDFy29DgZ5Dqn
+flAtbbEyVG+VCwwhDgUT+FTNNS1eg18GStr6LNECgYASgo1anUgbhax0wa5V38xR
+e8AUcJyFQ+Ns4q03DV2pNMAIc9Fqr2IsQVcaG0iRJlE8hqzV0AU8mGUmWI30Exbc
+QS2a+mIZyOQst/VwoX2sfI5WDrwdGB2XLrHv/Qmn9euehhESP21RJMTOYm2yDD8P
+GUxo/tcTAtKexbuJn5VyoQKBgB7OH0DhmZvAlOWdCgc9P20hMURZBwhZLFDIqAjT
+2EPIIRJuK+nuG/DUcb7b7OalixRMJtt9Nly4jhKD/ChzOmgFlI9L0EuzLM0YIyFk
+2cPFxt6Pxef+DuR6fKN+1oegNstSwx8cAfPkNh1QbBcmLQXiaUeGWnmgTGoZQFP5
+65eBAoGAfV98Mwka+VJ3hYNPL2ZHUXHnXw9Hnf5NnaGfgz7/Ucw3H88HsrIDIZgO
+NKSM3NVRIrweAx8/gDIrGqjXkvrwuCqXXYeS23gRteigUpoQrtGjBxIwtalT8K2O
+jI4vqz8SsNALtR8nehmBPzTj+t+rF5b1cMfyreHccoAa+0TbPac=
+-----END RSA PRIVATE KEY-----