Service creation delay when creating many deployments in a short time

[aii-nozomu-oki]

How to reproduce

Skupper instalation

kubectl create namespace app
kubectl config set-context --current --namespace app
kubectl apply -f https://raw.githubusercontent.com/skupperproject/skupper/1.3.0/cmd/site-controller/deploy-watch-current-ns.yaml

Create skupper-site.yaml

apiVersion: v1
data:
  console: "false"
  console-authentication: internal
  console-password: ""
  console-user: ""
  flow-collector: "false"
  ingress: none
  name: app
  router-console: "false"
  router-logging: ""
  router-mode: edge
  service-controller: "true"
  service-sync: "true"
kind: ConfigMap
metadata:
  name: skupper-site
  namespace: app

kubectl apply -f skupper-site.yaml

Run the following script to create 100 annotated deployments

#!/bin/bash

function create_service () {
    name="$(cat /dev/urandom | tr -dc 'a-z' | fold -w 16 | head -n 1)"
    deploy="
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${name}
  labels:
    app: ${name}
    env: test
  annotations:
    skupper.io/proxy: \"http\"
    skupper.io/port: \"8080:80\"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ${name}
  template:
    metadata:
      labels:
        app: ${name}
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80"
    echo "${deploy}" | kubectl apply -f -
}

for i in $(seq 0 99)
do
    create_service
    sleep 1
done

If I watch services, I will see that there is a delay.
Is this expected behavior?

Skupper version

1.3.0

Kubernetes version

Both in K3s v1.27.1+k3s1 and GKE v1.25.8-gke.500, this behavior is confirmed

Related issue

This may be related to #915, as it became apparent while researching it.

[aii-nozomu-oki]

Thanks!