Skupper 1.5.5 skupper-network-status configMap with inconsistent linkCost values #1654

johnnally · 2024-09-13T11:51:53Z

Describe the bug
I am deploying skupper 1.5.5 with the gRPC Online Boutique demo and am noticing that the link cost displayed for the links in the skupper-network-status configMap is inconsistent. Sometimes it is the same as seen via the skupper link status <link-name> -v
output and at other times it is different. The console api seems to be consistent with the skupper link status <link-name> -v but the collectorlite(skupper-network-status) configMap is not.

How To Reproduce
I used below steps to reproduce:
1. Install Online Boutique deployments
CLUSTERA

export OHN="frontend"
kubectl create ns $OHN
kubectl config set-context --current --namespace $OHN
git clone https://github.com/skupperproject/skupper-example-grpc.git
kubectl create -f skupper-example-grpc/deployment-ms-a.yaml
kubectl expose deployment productcatalogservice --port 3550
kubectl expose deployment recommendationservice --port 8080

CLUSTERB

export OHN="backend"
kubectl create ns $OHN
kubectl config set-context --current --namespace $OHN
git clone https://github.com/skupperproject/skupper-example-grpc.git
kubectl create -f skupper-example-grpc/deployment-ms-b.yaml
kubectl expose deployment checkoutservice --port 5050
kubectl expose deployment cartservice --port 7070
kubectl expose deployment currencyservice --port 7000
kubectl expose deployment adservice --port 9555
kubectl expose deployment redis-cart --port 6379

CLUSTERC

export OHN="backend2"
kubectl create ns $OHN
kubectl config set-context --current --namespace $OHN
git clone https://github.com/skupperproject/skupper-example-grpc.git
kubectl create -f skupper-example-grpc/deployment-ms-c.yaml
kubectl expose deployment emailservice --port 5000 --target-port 8080
kubectl expose deployment paymentservice --port 50051
kubectl expose deployment shippingservice --port 50051

2. Install skupper v1.5.5 on all 3 clusters

mkdir /root/skupper-1.5/
wget https://github.com/skupperproject/skupper/releases/download/1.5.5/skupper-cli-1.5.5-linux-amd64.tgz
tar zxvf skupper-cli-1.5.5-linux-amd64.tgz
chmod 775 skupper

3. Setup skupper links
CLUSTERB (backend) -> please note this is clusterB first and not clusterA

cd /root/skupper-1.5/
# from v1.5 we need to explicity enable console with skupper init
./skupper init --site-name clusterB --enable-console --enable-flow-collector -n $OHN
./skupper token create clusterB-token.yaml --uses 2 -n $OHN
cat  clusterB-token.yaml
# install site controller ( i downloaded the site controller yaml from skupper github somwehere ? not sure if it is needed or not to reproduce this bug )
kubectl apply -f ~/sc.yaml -n $OHN

# scp clusterB-token.yaml to clusterA and clusterC
scp clusterB-token.yaml root@<CLUSTERA>:
scp clusterB-token.yaml root@<CLUSTERC>:

CLUSTERC (backend2)

cd /root/skupper-1.5/
./skupper init --site-name clusterC --enable-console --enable-flow-collector -n $OHN
./skupper token create clusterC-token.yaml -n $OHN
cat clusterC-token.yaml
./skupper link create ~/clusterB-token.yaml -n $OHN
# install site controller
kubectl apply -f ~/sc.yaml -n $OHN

# scp clusterC-token.yaml to clusterA
scp clusterC-token.yaml root@<CLUSTERA>:

CLUSTERA (frontend)

cd /root/skupper-1.5/
./skupper init --site-name clusterA --enable-console --enable-flow-collector -n $OHN
./skupper link create /root/clusterB-token.yaml -n $OHN
./skupper link create /root/clusterC-token.yaml -n $OHN
# install site controller
kubectl apply -f ~/sc.yaml -n $OHN

RemoteCconnection Directions:
clusterA -> clusterC
clusterA -> clusterB
clusterC -> clusterB

4. Verify Skupper CLI link costs via skupper cli
CLUSTERA (frontend)

$ ./skupper link status -n $OHN

Links created from this site:

         Link link1 is connected
         Link link2 is connected

Current links from other sites that are connected:

         There are no connected links

$ ./skupper link status link1 -v -n $OHN

 Cost:          1
 Created:       2024-09-13 03:41:15 -0700 PDT
 Name:          link1
 Namespace:     frontend
 Site:          clusterA-ab1c3c6c-6c95-4a62-9ed5-85284760b759
 Status:        Connected

$ ./skupper link status link2 -v -n $OHN

 Cost:          1
 Created:       2024-09-13 03:41:15 -0700 PDT
 Name:          link2
 Namespace:     frontend
 Site:          clusterA-ab1c3c6c-6c95-4a62-9ed5-85284760b759
 Status:        Connected

Note how each of the above outgoing links(from clusterA to clusterB/clusterC) have a Cost of 1

CLUSTERB (backend)

$ ./skupper link status -n $OHN

Links created from this site:

         There are no links configured or connected

Current links from other sites that are connected:

         Incoming link from site 4c7184d4-f4d6-4faa-8a1f-f3d483dda195 on namespace frontend

CLUSTERC (backend2)

$ ./skupper link status -n $OHN

Links created from this site:

         There are no links configured or connected

Current links from other sites that are connected:

         Incoming link from site 4c7184d4-f4d6-4faa-8a1f-f3d483dda195 on namespace frontend


$ ./skupper link status link1 -v -n $OHN

 Cost:          1
 Created:       2024-09-13 03:40:20 -0700 PDT
 Name:          link1
 Namespace:     backend2
 Site:          clusterC-c045fc7d-44ca-4755-970d-6ab4b6c9ccee
 Status:        Connected

Status: Connected

Again a cost of 1 for the outgoing link rom clusterC->clusterB

5. Expose Demo services
CLUSTERA (frontend)

/root/skupper-example-grpc/expose-deployments-a.sh

CLUSTERB (backend)

/root/skupper-example-grpc/expose-deployments-b.sh

CLUSTERC (backend2)

/root/skupper-example-grpc/expose-deployments-c.sh

6. Log into Demo Ui and make sure the UI is working and generate some traffic
check frontend-external IP:
$ kubectl get svc frontend-external -o jsonpath='{ .status.loadBalancer.ingress}'
[{"ip":""}]

http:///

7. Check API calls to the collector API and investigate links

CLUSTERA
links found by collector API

		{
			"recType": "LINK",
			"identity": "bds7j:1",
			"parent": "bds7j:0",
			"startTime": 1726224079428097,
			"endTime": 0,
			"source": "bds7j:0",
			"mode": "interior",
			"name": "clusterB-skupper-router-86bb8f95fb-84q62",
			"direction": "outgoing"
		},
		{
			"recType": "LINK",
			"identity": "bds7j:2",
			"parent": "bds7j:0",
			"startTime": 1726224087474461,
			"endTime": 0,
			"source": "bds7j:0",
			"mode": "interior",
			"name": "clusterC-skupper-router-7fdcb4f9c5-gzvwm",
			"direction": "outgoing"
		}

Note how no links have a cost associated with them.

links seen in skupper-network-status configMap
As a result when I parse the skupper-network-status configMap with jq I can see no link Cost:

          "links": [
            {
              "recType": "LINK",
              "identity": "bds7j:1",
              "parent": "bds7j:0",
              "startTime": 1726224079428097,
              "endTime": 0,
              "source": "bds7j:0",
              "mode": "interior",
              "name": "clusterB-skupper-router-86bb8f95fb-84q62",
              "direction": "outgoing"
            },
            {
              "recType": "LINK",
              "identity": "bds7j:2",
              "parent": "bds7j:0",
              "startTime": 1726224087474461,
              "endTime": 0,
              "source": "bds7j:0",
              "mode": "interior",
              "name": "clusterC-skupper-router-7fdcb4f9c5-gzvwm",
              "direction": "outgoing"
            }
          ],

See full skupper-network-status configMap for clusterA below.
clusterA-skupper-network-status.txt

links found by console API

	{
		"Name": "link1",
		"Url": "9.20.195.15:55671",
		"Cost": 1,
		"Connected": true,
		"Configured": true,
		"Description": "[C27] Connection Opened: dir=out host=9.20.195.15:55671 encrypted=TLSv1.3 auth=EXTERNAL user=(null) container_id=clusterB-skupper-router-86bb8f95fb-84q62",
		"Created": "2024-09-13T10:41:15Z"
	},
	{
		"Name": "link2",
		"Url": "9.30.26.214:55671",
		"Cost": 1,
		"Connected": true,
		"Configured": true,
		"Description": "[C43] Connection Opened: dir=out host=9.30.26.214:55671 encrypted=TLSv1.3 auth=EXTERNAL user=(null) container_id=clusterC-skupper-router-7fdcb4f9c5-gzvwm",
		"Created": "2024-09-13T10:41:15Z"
	}

Note how the console API shows the link Cost for both of clusterA's outgoing links.

It seems to be a bug that the collectorAPI does not always match the linkCost values that are seen via the skupper cli or the console API

CLUSTERB
links found by collector API

		{
			"recType": "LINK",
			"identity": "84q62:1",
			"parent": "84q62:0",
			"startTime": 1726224026429955,
			"endTime": 0,
			"source": "84q62:0",
			"mode": "interior",
			"name": "clusterC-skupper-router-7fdcb4f9c5-gzvwm",
			"direction": "incoming"
		},
		{
			"recType": "LINK",
			"identity": "84q62:2",
			"parent": "84q62:0",
			"startTime": 1726224080376641,
			"endTime": 0,
			"source": "84q62:0",
			"mode": "interior",
			"name": "clusterA-skupper-router-5d5d57954f-bds7j",
			"linkCost": 1,
			"direction": "incoming"
		}

links seen in skupper-network-status configMap

            {
              "recType": "LINK",
              "identity": "84q62:1",
              "parent": "84q62:0",
              "startTime": 1726224026429955,
              "endTime": 0,
              "source": "84q62:0",
              "mode": "interior",
              "name": "clusterC-skupper-router-7fdcb4f9c5-gzvwm",
              "direction": "incoming"
            },
            {
              "recType": "LINK",
              "identity": "84q62:2",
              "parent": "84q62:0",
              "startTime": 1726224080376641,
              "endTime": 0,
              "source": "84q62:0",
              "mode": "interior",
              "name": "clusterA-skupper-router-5d5d57954f-bds7j",
              "linkCost": 1,
              "direction": "incoming"
            }

links found by console API
-> empty list as expected as both links are incoming to clusterB

For clusterB again an inconsistency is seen, both links are incoming but only one show a linkCost via the collector API/skupper-network-status configMap.
You would expect both to show a linkCost (or none as the links are incoming ), but one incoming link has a linkCost and the other doesn't.

CLUSTERC
links found by collector API

		{
			"recType": "LINK",
			"identity": "gzvwm:1",
			"parent": "gzvwm:0",
			"startTime": 1726224027445065,
			"endTime": 0,
			"source": "gzvwm:0",
			"mode": "interior",
			"name": "clusterB-skupper-router-86bb8f95fb-84q62",
			"direction": "outgoing"
		},
		{
			"recType": "LINK",
			"identity": "gzvwm:2",
			"parent": "gzvwm:0",
			"startTime": 1726224086458022,
			"endTime": 0,
			"source": "gzvwm:0",
			"mode": "interior",
			"name": "clusterA-skupper-router-5d5d57954f-bds7j",
			"direction": "incoming"
		}

links seen in skupper-network-status configMap

          "links": [
            {
              "recType": "LINK",
              "identity": "gzvwm:1",
              "parent": "gzvwm:0",
              "startTime": 1726224027445065,
              "endTime": 0,
              "source": "gzvwm:0",
              "mode": "interior",
              "name": "clusterB-skupper-router-86bb8f95fb-84q62",
              "direction": "outgoing"
            },
            {
              "recType": "LINK",
              "identity": "gzvwm:2",
              "parent": "gzvwm:0",
              "startTime": 1726224086458022,
              "endTime": 0,
              "source": "gzvwm:0",
              "mode": "interior",
              "name": "clusterA-skupper-router-5d5d57954f-bds7j",
              "direction": "incoming"
            }

links found by console API

	{
		"Name": "link1",
		"Url": "9.20.195.15:55671",
		"Cost": 1,
		"Connected": true,
		"Configured": true,
		"Description": "[C26] Connection Opened: dir=out host=9.20.195.15:55671 encrypted=TLSv1.3 auth=EXTERNAL user=(null) container_id=clusterB-skupper-router-86bb8f95fb-84q62",
		"Created": "2024-09-13T10:40:20Z"
	}

For clusterC the console API/skupper cli report a link cost of 1 for the outgoing link from clusterC->clusterB.
But this linkCost is not reported by the collector(lite) API/skupper-network-status configMap.

Expected behavior
You would expect the linkCost for link detail returned by the flow collector(lite) api/skupper-network-status configMap to always match what is seen in the console api and skupper cli outputs when querying the link details.
There seems to be a bug in the way the linkCost is displayed in the skupper-network-status configMap/collector(lite) API

Environment details

Skupper CLI: 1.5.5
Platform: kubernetes

./skupper version
client version                 1.5.5
transport version              quay.io/skupper/skupper-router:2.5.3 (sha256:51b3a4549b7b)
controller version             quay.io/skupper/service-controller:1.5.5 (sha256:1060c4267927)
config-sync version            quay.io/skupper/config-sync:1.5.5 (sha256:8ea9f26d06fa)
flow-collector version         quay.io/skupper/flow-collector:1.5.5 (sha256:3b5f91e3f4b9)

Additional context
Please see below skupper debug dumps
clusterA.tar.gz
clusterB.tar.gz
clusterC.tar.gz

The text was updated successfully, but these errors were encountered:

johnnally · 2024-09-13T12:58:03Z

Just to add this affects how Hybrid Cloud Mesh processes brownfield 1.5.X+ Gateways.
As a result we not be able to get the correct linkCost via the skupper-network-status api

lynnemorrison self-assigned this Sep 27, 2024

lynnemorrison mentioned this issue Oct 1, 2024

Update link cost when receiving a flow record #1695

Merged

ajssmith closed this as completed in #1695 Oct 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skupper 1.5.5 skupper-network-status configMap with inconsistent linkCost values #1654

Skupper 1.5.5 skupper-network-status configMap with inconsistent linkCost values #1654

johnnally commented Sep 13, 2024 •

edited

Loading

johnnally commented Sep 13, 2024

Skupper 1.5.5 skupper-network-status configMap with inconsistent linkCost values #1654

Skupper 1.5.5 skupper-network-status configMap with inconsistent linkCost values #1654

Comments

johnnally commented Sep 13, 2024 • edited Loading

johnnally commented Sep 13, 2024

johnnally commented Sep 13, 2024 •

edited

Loading