Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Federation Between VMs and Kubernetes ACL Setting issue #10138

Closed
ZEROYXY opened this issue Apr 28, 2021 · 2 comments
Closed

Federation Between VMs and Kubernetes ACL Setting issue #10138

ZEROYXY opened this issue Apr 28, 2021 · 2 comments
Labels
theme/acls ACL and token generation theme/connect Anything related to Consul Connect, Service Mesh, Side Car Proxies theme/mesh-gw Track mesh gateway work type/bug Feature does not function as expected

Comments

@ZEROYXY
Copy link

ZEROYXY commented Apr 28, 2021 •
edited by jsosulska
Loading

Hi all, I am facing an issue when I am trying to build the Federation of Consul Mesh-Gateway. I have been building four consul clusters and make them accross each other by the Consul Mesh-Geteway. I built the primary consul cluster on the Kubernetes and it has been running well since it started. Then I added two consul clusters which built on the K8s to make them accoss each other by the Consul Mesh-Gateway Federation.
I have been tried to add a consul cluster which built on VM into Federation mentioned above and I followed the steps of https://www.consul.io/docs/k8s/installation/multi-cluster/vms-and-kubernetes ---- Kubernetes as the Primary. Then I faced the issue as below:

2021-04-28T15:25:18.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.77:8302: Remote DC has no server currently reachable
2021-04-28T15:25:18.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.146:8302: Remote DC has no server currently reachable
2021-04-28T15:25:18.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.207:8302: Remote DC has no server currently reachable
2021-04-28T15:25:19.116+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.100.186.199:8302: Remote DC has no server currently reachable
2021-04-28T15:25:19.116+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.77:8302: Remote DC has no server currently reachable
2021-04-28T15:25:19.116+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.100.140.78:8302: Remote DC has no server currently reachable
2021-04-28T15:25:19.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.146:8302: Remote DC has no server currently reachable
2021-04-28T15:25:19.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.100.186.199:8302: Remote DC has no server currently reachable
2021-04-28T15:25:26.616+0800 [INFO]  agent.server.memberlist.wan: memberlist: Suspect consul-server-0.dc1 has failed, no acks received
2021-04-28T15:25:27.082+0800 [INFO]  agent.server.serf.lan: serf: EventMemberUpdate: consul-dc4-server2
2021-04-28T15:25:27.082+0800 [INFO]  agent.server: Updating LAN server: server="consul-dc4-server2 (Addr: tcp/192.168.20.71:8300) (DC: dc4)"
2021-04-28T15:25:27.116+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.100.140.78:8302: Remote DC has no server currently reachable
2021-04-28T15:25:27.117+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.100.186.199:8302: Remote DC has no server currently reachable
2021-04-28T15:25:27.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.146:8302: Remote DC has no server currently reachable
2021-04-28T15:25:27.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.100.186.199:8302: Remote DC has no server currently reachable
2021-04-28T15:25:27.881+0800 [INFO]  agent.server.serf.wan: serf: EventMemberUpdate: consul-dc4-server2.dc4
2021-04-28T15:25:27.882+0800 [INFO]  agent.server: Handled event for server in area: event=member-update server=consul-dc4-server2.dc4 area=wan
2021-04-28T15:25:28.116+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.207:8302: Remote DC has no server currently reachable
2021-04-28T15:25:28.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.146:8302: Remote DC has no server currently reachable
2021-04-28T15:25:28.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.100.248.252:8302: Remote DC has no server currently reachable
2021-04-28T15:25:28.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.100.186.199:8302: Remote DC has no server currently reachable
2021-04-28T15:25:29.115+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.100.248.252:8302: Remote DC has no server currently reachable
2021-04-28T15:25:29.116+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.207:8302: Remote DC has no server currently reachable
2021-04-28T15:25:31.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: Remote DC has no server currently reachable
2021-04-28T15:25:36.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: Remote DC has no server currently reachable
2021-04-28T15:25:41.494+0800 [ERROR] agent: Coordinate update error: error="ACL not found"
2021-04-28T15:25:41.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: Remote DC has no server currently reachable
2021-04-28T15:25:46.396+0800 [ERROR] agent.server.memberlist.wan: memberlist: Push/Pull with consul-server-0.dc2 failed: Remote DC has no server currently reachable
2021-04-28T15:25:46.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: Remote DC has no server currently reachable
2021-04-28T15:25:51.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: Remote DC has no server currently reachable
2021-04-28T15:25:56.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send ping: Remote DC has no server currently reachable
2021-04-28T15:25:58.422+0800 [ERROR] agent: Coordinate update error: error="ACL not found"
2021-04-28T15:25:59.905+0800 [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found"
2021-04-28T15:26:04.617+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send indirect ping: Remote DC has no server currently reachable
2021-04-28T15:26:13.443+0800 [ERROR] agent: Coordinate update error: error="ACL not found"
2021-04-28T15:26:22.050+0800 [ERROR] agent.anti_entropy: failed to sync remote state: error="ACL not found"
2021-04-28T15:26:27.116+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.146:8302: Remote DC has no server currently reachable
2021-04-28T15:26:27.116+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.100.186.199:8302: Remote DC has no server currently reachable
2021-04-28T15:26:27.616+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.207:8302: Remote DC has no server currently reachable
2021-04-28T15:26:28.116+0800 [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 173.0.0.207:8302: Remote DC has no server currently reachable

The configuration of the /etc/consul.d/consul.hcl is as below:

cert_file = "/home/cloud/consul/dc4-server-consul-0.pem"
key_file = "/home/cloud/consul/dc4-server-consul-0-key.pem"
ca_file = "/home/cloud/consul/consul-agent-ca.pem"
primary_gateways = ["192.168.**.**:443"]
acl {
  enabled = true
  default_policy = "deny"
  down_policy = "extend-cache"
  tokens {
    agent = "a6c61787-e229-41f7-8541-0e5adc414b34"
    replication = "05eea11e-f7e8-9635-de3e-a8c9d1439135"
  }
}
encrypt = "0NcEVIpbnovNODzVnPSXo0QnuLuNXXjkmKzpVwHnX4E="

# Other server settings
server = true
datacenter = "dc4"
data_dir = "/opt/consul"
enable_central_service_config = true
primary_datacenter = "dc1"
connect {
  enabled = true
  enable_mesh_gateway_wan_federation = true
}
verify_incoming_rpc = true
verify_outgoing = true
verify_server_hostname = true
ports {
  https = 8501
  http = -1
  grpc = 8502
}

log_level = "INFO"
node_name = "consul-dc4-server1"
bind_addr = "192.168.**.**"

I got the consul-gossip-encryption-key from K8s as below
kubectl get secrets/consul-gossip-encryption-key --template='{{.data.key}}' |base64 -d
Is what I did as above is currect?

I got the consul-acl-replication-acl-token from K8s as below

kubectl get secrets/consul-acl-replication-acl-token --template='{{.data.token}}' |base64 -d
05eea11e-f7e8-9635-de3e-a8c9d1439135

The command used on the website without the parameter base64 -d. But the token got in that way is strange and show as below:

[root@consul-dc1-master consul]# kubectl get secrets/consul-acl-replication-acl-token --template='{{.data.token}}'
MDVlZWExMWUtZjdlOC05NjM1LWRlM2UtYThjOWQxNDM5MTM1

I don't think the token MDVlZWExMWUtZjdlOC05NjM1LWRlM2UtYThjOWQxNDM5MTM1 should be the correct one, right?

Then I didn't understand where I can find the agent = "<your agent token>" on the website https://www.consul.io/docs/k8s/installation/multi-cluster/vms-and-kubernetes as below

acls {
  tokens {
    agent = "<your agent token>"
    replication = "e7924dd1-dc3f-f644-da54-81a73ba0a178"
  }
}

I set the part of ACL as below but it dose not work. I doubt if it is caused by the set of agent = "a6c61787-e229-41f7-8541-0e5adc414b34" . Could you please kindly tell from where I can get the exactly agent token and shall I use the primary consul cluster agent token or the Secondry VM Consul Cluster agent token?

acl {
  enabled = true
  default_policy = "deny"
  down_policy = "extend-cache"
  tokens {
    agent = "a6c61787-e229-41f7-8541-0e5adc414b34"
    replication = "05eea11e-f7e8-9635-de3e-a8c9d1439135"
  }
}
@ChipV223 ChipV223 mentioned this issue Apr 28, 2021
Closed
@ChipV223
Copy link
Contributor

Hi @ZEROYXY !

Could you please kindly tell from where I can get the exactly agent token and shall I use the primary consul cluster agent token or the Secondry VM Consul Cluster agent token?

For this VM cluster, you would actually need to create an Agent policy and token and use that in the config. This is actually discussed in the ACL section of this page in the Consul documentation

As far as the Remote DC has no server currently reachable errors, it seems like the network settings for the VM cluster have not properly set up to reached the K8s clusters so you would want to resolve those in order for the VM cluster to join the K8s clusters

@jsosulska jsosulska added theme/acls ACL and token generation theme/connect Anything related to Consul Connect, Service Mesh, Side Car Proxies theme/mesh-gw Track mesh gateway work type/bug Feature does not function as expected labels May 10, 2021
@ChipV223 ChipV223 added the waiting-reply Waiting on response from Original Poster or another individual in the thread label May 26, 2021
@jsosulska
Copy link
Contributor

jsosulska commented Jun 7, 2021 •
edited
Loading

No response in approximately 6 weeks. Closing due to no response.
A docs update has been done in #10233 . to address why the agent token is need.

@ghost ghost removed waiting-reply Waiting on response from Original Poster or another individual in the thread labels Jun 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
theme/acls ACL and token generation theme/connect Anything related to Consul Connect, Service Mesh, Side Car Proxies theme/mesh-gw Track mesh gateway work type/bug Feature does not function as expected
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ChipV223 @jsosulska @ZEROYXY