Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Facing issue in submariner gateway. #3188

Closed
Nitish-CONSORT opened this issue Oct 17, 2024 · 20 comments · Fixed by submariner-io/lighthouse#1675
Closed

Facing issue in submariner gateway. #3188

Nitish-CONSORT opened this issue Oct 17, 2024 · 20 comments · Fixed by submariner-io/lighthouse#1675
Assignees
@yboaron
Labels
need-info priority:low support

Comments

@Nitish-CONSORT
Copy link

Nitish-CONSORT commented Oct 17, 2024 •
edited
Loading

I have a submariner setup in which 2 cluster are connected, cluster1 and broker is running on same host machine.

I am facing a issue during host machine restart:
Issue: when I am checking status by "subctl show connections", it is giving output "connected" state and all submariner pods are running properly but after checking gateways pod logs it is showing "not found any active active connection" and even exported services are not showing on another cluster as imported services.

subctl show connections
Cluster "kubernetes"
✓ Showing Connections
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
consort microk8s-cluster 192.168.1.51 no libreswan 172.30.183.0/24, 172.17.0.0/16 connected 46.848µs

Logs:
.168.2.6 --client 172.16.0.0/16 --ikeport 4500 --to --id 192.168.1.51 --host 192.168.1.51 --client 172.17.0.0/16 --ikeport 4500 --dpdaction=hold --dpddelay 30]
002 "submariner-cable-microk8s-cluster-192-168-1-51-1-1": added IKEv2 connection
2024-10-16T10:35:47.798Z INF ..gine/cableengine.go:226 CableEngine Successfully installed Endpoint cable "submariner-cable-microk8s-cluster-192-168-1-51" with remote IP 192.168.1.51
2024-10-16T10:35:51.117Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:35:51.117Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:35:51.117Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:35:51.117Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:35:51.117Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:35:56.173Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:35:56.173Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:35:56.173Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:35:56.173Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:35:56.173Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:01.241Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:01.241Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:01.241Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:01.241Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:01.241Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:06.298Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:06.298Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:06.298Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:06.298Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:06.298Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:11.374Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:11.374Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:11.374Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:11.374Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:11.374Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:16.401Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:16.401Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:16.401Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:16.401Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:16.401Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:21.458Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:21.458Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:21.458Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:21.458Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:21.458Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:26.526Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:26.526Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:26.526Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:26.526Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:26.526Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:31.579Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:31.579Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:31.579Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:31.579Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:31.579Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:36.601Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:36.601Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:36.601Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:36.601Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:36.601Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:41.625Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:41.625Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:41.625Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:41.625Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:41.625Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:46.652Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:46.652Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:46.652Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:46.652Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:46.652Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:51.675Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:51.675Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:51.675Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:51.675Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:51.675Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:56.700Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:56.700Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:56.700Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:56.700Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:36:56.700Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:01.786Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:01.786Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:01.786Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:01.786Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:01.786Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:06.808Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:06.808Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:06.808Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:06.809Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:06.809Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:11.841Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:11.841Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:11.841Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:11.841Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:11.841Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:16.896Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:16.896Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:16.896Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:16.896Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:16.896Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:21.919Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:21.919Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:21.919Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:21.919Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:21.919Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:27.046Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:27.046Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:27.046Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:27.046Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:27.046Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:32.096Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:32.096Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:32.096Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:32.096Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:32.096Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:37.147Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:37.147Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:37.147Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:37.147Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:37.147Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:42.198Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:42.198Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:42.198Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:42.198Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:42.198Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:47.256Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:47.256Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:47.256Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:47.256Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:47.256Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:52.306Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:52.306Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:52.306Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:52.306Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:52.306Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:57.423Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:57.423Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:57.423Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:57.423Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:37:57.423Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:02.482Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:02.482Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:02.482Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:02.482Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:02.482Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:07.738Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:07.738Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:07.739Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:07.739Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:07.739Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:12.847Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:12.847Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:12.847Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:12.847Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:12.847Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:17.867Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:17.867Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:17.867Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:17.867Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:17.868Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:22.894Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:22.894Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:22.894Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:22.894Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:22.894Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:27.953Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:27.953Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:27.953Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:27.953Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:27.953Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:32.975Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:32.975Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:32.975Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:32.975Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:32.975Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:38.034Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:38.034Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:38.034Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:38.034Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:38.034Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:43.091Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:43.091Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:43.091Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:43.091Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:43.091Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:48.166Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:48.166Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:48.166Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:48.167Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:48.167Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:53.219Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:53.219Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:53.219Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:53.219Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:53.219Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:58.300Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:58.300Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:58.300Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:58.300Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:38:58.300Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:03.352Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:03.352Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:03.352Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:03.352Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:03.352Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:08.394Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:08.394Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:08.394Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:08.394Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:08.394Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:13.463Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:13.463Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:13.463Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:13.463Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:13.463Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:18.545Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:18.545Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:18.545Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:18.545Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:18.545Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:23.667Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:23.668Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:23.668Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:23.668Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:23.668Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:28.722Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:28.722Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:28.722Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:28.722Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:28.722Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:33.814Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:33.814Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:33.814Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:33.814Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:33.814Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:38.905Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:38.905Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:38.905Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:38.905Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:38.905Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:44.076Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:44.076Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:44.076Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:44.076Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:44.076Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:49.139Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:49.139Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-0-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:49.139Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-0" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:49.139Z DBG ..reswan/libreswan.go:268 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51-1-1" not found in active connections obtained from whack: map[], map[]
2024-10-16T10:39:49.139Z DBG ..reswan/libreswan.go:282 libreswan Connection "submariner-cable-microk8s-cluster-192-168-1-51" not found in active connections obtained from whack: map[], map[]
@github-project-automation github-project-automation bot moved this to Backlog in Backlog Oct 22, 2024
@dfarrell07
Copy link
Member

@Nitish-CONSORT Can you provide subctl show and subctl diagnose outputs so we can see more of what's going on?

@Nitish-CONSORT
Copy link
Author

dianose all
subctl all

@tpantelis
Copy link
Contributor

Some things of note:

  • The lighthouse agent and coredns pods have restarted 8 times. That could be b/c you've restarted them manually or there was an issue which caused the process to prematurely exit and restart.

  • It couldn't conclusively verify if the Calico IPPool is configured with VXLAN overlay encapsulation. I'm not familiar with Calico so I don't know if that's an issue or not.

  • It reports that a service named "idm-service" was exported but it's of type NodePort which we don't support.

The provided output was from the "kubernetes" cluster which also seems to be the broker cluster. Can you run the same commands on the other cluster?

It's unclear why "ipsec whack" reports that 192-168-1-51 is not found in the active connections. Are you seeing a connectivity issue between the clusters? Have you tried running subctl verify?

@yboaron
Copy link
Contributor

yboaron commented Oct 28, 2024 •
edited
Loading

@Nitish-CONSORT ,
It could be an issue similar to #2571 ,
can you try restarting Submariner GW and RouteAgent pods on both clusters and see if that helps?

You can use
kubectl delete pods -n submariner-operator -l app=submariner-routeagent

kubectl delete pods -n submariner-operator -l app=submariner-gateway

@Nitish-CONSORT
Copy link
Author

Yes restart works for me but I can't do restart to restore system in production environment. Please suggest steps to restore system automatically.

@yboaron
Copy link
Contributor

yboaron commented Oct 29, 2024

but I can't do restart to restore system in production environment. Please suggest steps to restore system automatically.

Understood,
A.
We've seen cases in the past where restarting the machine host caused Submariner generated IP rules/routes to be deleted on the k8s node (which is actually a container or VM in your case) host network.

Each K8S node runs a Kubelet and container runtime (e.g. CRI-O), the expected behavior for restarting the machine host depends on how the container runtime handles restart. If when a machine host is restarted, the container runtime (on the K8S node) stops all its containers and starts them again, then I would expect all Submariner pods to be restarted and traffic between clusters should be fine.

If when restarting a machine host, the container runtime suspends/resumes its containers and doesn't restart them it could break traffic between clusters, B/C Submariner pods like all other pods on the node are not restarted.

If it is the second case in your environment, I don't think there is anything to do on Submariner side but maybe in the container runtime settings

Can you check how container runtime in K8S node behaves following machine host restart in your env?

B. Can you elaborate on your environment? how do you apply host machine restart?

C. Please attach submariner logs from both clusters (you can use subctl gather --kubeconfig < kubeconfig> ) after you apply host restart

@yboaron
Copy link
Contributor

yboaron commented Nov 12, 2024

@Nitish-CONSORT Any update on this issue?

@Nitish-CONSORT
Copy link
Author

submariner gateway issue has been resolved.
But still i am facing service export issues during restart of broker host. Means service export is not showing as imported service on other cluster after restart of broker, cluster1 host.

@yboaron
Copy link
Contributor

yboaron commented Nov 12, 2024

submariner gateway issue has been resolved. But still i am facing service export issues during restart of broker host. Means service export is not showing as imported service on other cluster after restart of broker, cluster1 host.

Could be the same root cause, did you try deleting submariner-lighthouse-agent and submariner-lighthouse-coredns pods ?

@dfarrell07 dfarrell07 removed this from Backlog Nov 12, 2024
@dfarrell07 dfarrell07 moved this to In Progress in Submariner 0.20 Nov 12, 2024
@Nitish-CONSORT
Copy link
Author

Nitish-CONSORT commented Nov 13, 2024 •
edited
Loading

I have deleted submariner-lighthouse-agent and submariner-lighthouse-coredns pods but still output is below

exportissue

@Nitish-CONSORT
Copy link
Author

Nitish-CONSORT commented Nov 13, 2024 •
edited
Loading

submariner-lighthouse-coredns log is below:

coredns-log

@tpantelis
Copy link
Contributor

Please run subctl gather and attach the files.

@Nitish-CONSORT
Copy link
Author

All files are attached.

submariner-20241113110317.zip

@tpantelis
Copy link
Contributor

I see from the submariner-lighthouse-agent log that the aggregated ServiceImport was created on the broker:

[90m2024-11-13T08:54:35.678Z�[0m DBG ..r/service_import.go:366 agent                Created aggregated ServiceImport {
  "metadata": {
    "name": "idms-service-default",
    "creationTimestamp": null,
    "annotations": {
      "lighthouse.submariner.io/sourceNamespace": "default",
      "multicluster.kubernetes.io/service-name": "idms-service"
    }
  },
  "spec": {
    "ports": [],
    "type": "ClusterSetIP"
  },
  "status": {
    "clusters": [
      {
        "cluster": "kubernetes"
      }
    ]
  }
}

But I don't see it in the gather output. That's strange. I wonder if it somehow got deleted. Can you attach subctl gather from the other cluster?

@Nitish-CONSORT
Copy link
Author

all logs of another node

submariner-20241113123016.zip

@tpantelis
Copy link
Contributor

I don't see anything of note. Try unexporting the service and then re-export.

@Nitish-CONSORT
Copy link
Author

I have un-exported and re-exported services again and it is working properly.
subctl gather logs of both nodes are attached below:

NODE-1-submariner-20241113132637.zip
NODE-2-submariner-20241113132542.zip

I have restarted the node 1 server and again issue start coming. subctl gather logs are attached below

NODE-1-submariner-20241113133837.zip
NODE-2-submariner-20241113133823.zip

@tpantelis
Copy link
Contributor

The ServiceImport is getting mysteriously deleted when the server node is restarted. Does this happen if you just restart the submariner-lighthouse-agent pod (after re-exporting the service)?

@Nitish-CONSORT
Copy link
Author

Yes, you are right. After restarting of submariner-lighthouse-agent pod, same issue is coming with service export.

tpantelis added a commit to tpantelis/lighthouse that referenced this issue Nov 14, 2024
@tpantelis
Fix inadvertent deletion of aggregated ServiceImports on agent restart
83f7f27 
If the broker co-exists on a managed cluster, on LH agent restart,
the aggregated ServiceImports on the broker are inadvertently deleted
during reconciliation. Reconciliation should only process local
aggregated ServiceImports and should ignore ServiceImports in the
broker namespace. The latter are distinguished by the presence of the
 "multicluster.kubernetes.io/service-name" annotation.

Fixes submariner-io/submariner#3188

Signed-off-by: Tom Pantelis <[email protected]>
tpantelis added a commit to tpantelis/lighthouse that referenced this issue Nov 14, 2024
@tpantelis
Fix inadvertent deletion of aggregated ServiceImports on agent restart
c299b2b 
If the broker co-exists on a managed cluster, on LH agent restart,
the aggregated ServiceImports on the broker are inadvertently deleted
during reconciliation. Reconciliation should only process local
aggregated ServiceImports and should ignore ServiceImports in the
broker namespace. The latter are distinguished by the presence of the
 "multicluster.kubernetes.io/service-name" annotation.

The reconciliation unit test was adjusted to cover this case.

Fixes submariner-io/submariner#3188

Signed-off-by: Tom Pantelis <[email protected]>
@tpantelis tpantelis mentioned this issue Nov 14, 2024
Merged
@tpantelis
Copy link
Contributor

I found the issue with inadvertent ServiceImport deletion and pushed a PR to fix it.

vthapar pushed a commit to submariner-io/lighthouse that referenced this issue Nov 15, 2024
@tpantelis @vthapar
Fix inadvertent deletion of aggregated ServiceImports on agent restart
68f3758 
If the broker co-exists on a managed cluster, on LH agent restart,
the aggregated ServiceImports on the broker are inadvertently deleted
during reconciliation. Reconciliation should only process local
aggregated ServiceImports and should ignore ServiceImports in the
broker namespace. The latter are distinguished by the presence of the
 "multicluster.kubernetes.io/service-name" annotation.

The reconciliation unit test was adjusted to cover this case.

Fixes submariner-io/submariner#3188

Signed-off-by: Tom Pantelis <[email protected]>
@github-project-automation github-project-automation bot moved this from In Progress to Done in Submariner 0.20 Nov 15, 2024
tpantelis added a commit to tpantelis/lighthouse that referenced this issue Nov 15, 2024
@tpantelis
Fix inadvertent deletion of aggregated ServiceImports on agent restart
60bf6aa 
If the broker co-exists on a managed cluster, on LH agent restart,
the aggregated ServiceImports on the broker are inadvertently deleted
during reconciliation. Reconciliation should only process local
aggregated ServiceImports and should ignore ServiceImports in the
broker namespace. The latter are distinguished by the presence of the
 "multicluster.kubernetes.io/service-name" annotation.

The reconciliation unit test was adjusted to cover this case.

Fixes submariner-io/submariner#3188

Signed-off-by: Tom Pantelis <[email protected]>
tpantelis added a commit to tpantelis/lighthouse that referenced this issue Nov 15, 2024
@tpantelis
Fix inadvertent deletion of aggregated ServiceImports on agent restart
ad15d54 
If the broker co-exists on a managed cluster, on LH agent restart,
the aggregated ServiceImports on the broker are inadvertently deleted
during reconciliation. Reconciliation should only process local
aggregated ServiceImports and should ignore ServiceImports in the
broker namespace. The latter are distinguished by the presence of the
 "multicluster.kubernetes.io/service-name" annotation.

The reconciliation unit test was adjusted to cover this case.

Fixes submariner-io/submariner#3188

Signed-off-by: Tom Pantelis <[email protected]>
tpantelis added a commit to tpantelis/lighthouse that referenced this issue Nov 15, 2024
@tpantelis
Fix inadvertent deletion of aggregated ServiceImports on agent restart
26ba441 
If the broker co-exists on a managed cluster, on LH agent restart,
the aggregated ServiceImports on the broker are inadvertently deleted
during reconciliation. Reconciliation should only process local
aggregated ServiceImports and should ignore ServiceImports in the
broker namespace. The latter are distinguished by the presence of the
 "multicluster.kubernetes.io/service-name" annotation.

The reconciliation unit test was adjusted to cover this case.

Fixes submariner-io/submariner#3188

Signed-off-by: Tom Pantelis <[email protected]>
skitt pushed a commit to submariner-io/lighthouse that referenced this issue Nov 15, 2024
@tpantelis @skitt
Fix inadvertent deletion of aggregated ServiceImports on agent restart
04864e9 
If the broker co-exists on a managed cluster, on LH agent restart,
the aggregated ServiceImports on the broker are inadvertently deleted
during reconciliation. Reconciliation should only process local
aggregated ServiceImports and should ignore ServiceImports in the
broker namespace. The latter are distinguished by the presence of the
 "multicluster.kubernetes.io/service-name" annotation.

The reconciliation unit test was adjusted to cover this case.

Fixes submariner-io/submariner#3188

Signed-off-by: Tom Pantelis <[email protected]>
skitt pushed a commit to submariner-io/lighthouse that referenced this issue Nov 15, 2024
@tpantelis @skitt
Fix inadvertent deletion of aggregated ServiceImports on agent restart
e0e429e 
If the broker co-exists on a managed cluster, on LH agent restart,
the aggregated ServiceImports on the broker are inadvertently deleted
during reconciliation. Reconciliation should only process local
aggregated ServiceImports and should ignore ServiceImports in the
broker namespace. The latter are distinguished by the presence of the
 "multicluster.kubernetes.io/service-name" annotation.

The reconciliation unit test was adjusted to cover this case.

Fixes submariner-io/submariner#3188

Signed-off-by: Tom Pantelis <[email protected]>
skitt pushed a commit to submariner-io/lighthouse that referenced this issue Nov 18, 2024
@tpantelis @skitt
Fix inadvertent deletion of aggregated ServiceImports on agent restart
3f59c25 
If the broker co-exists on a managed cluster, on LH agent restart,
the aggregated ServiceImports on the broker are inadvertently deleted
during reconciliation. Reconciliation should only process local
aggregated ServiceImports and should ignore ServiceImports in the
broker namespace. The latter are distinguished by the presence of the
 "multicluster.kubernetes.io/service-name" annotation.

The reconciliation unit test was adjusted to cover this case.

Fixes submariner-io/submariner#3188

Signed-off-by: Tom Pantelis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yboaron yboaron

Labels
need-info priority:low support
Projects
Submariner 0.20
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix inadvertent deletion of aggregated ServiceImports on agent restart tpantelis/lighthouse
5 participants
@dfarrell07 @tpantelis @yboaron @maayanf24 @Nitish-CONSORT