Failed to update listener port of a Gateway CR

[modatwork]

Description:

After updating the Gateway listener’s port, Envoy should listen on the new port and stop listening on the old port.

Reproduction Steps:

1. Set up the Envoy gateway and Custom Resources (CRs) by following the quickstart guide. Initial setup works fine.
2. Update the listener port in the Gateway Custom Resource (CR) to change it from 80 to 8001. After applying the updated Gateway CR, Envoy is not ready on either port 80 or 8001.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 8001

Environment:

gateway version: gateway-dev:latest (imageID: docker-pullable://envoyproxy/gateway-dev@sha256:4a84a34a7955e4cf7fc276725957c11283c515b2f1e93684d753fa0582e540ba)

Logs:

2024-04-07T08:29:53.884Z        INFO    provider        kubernetes/controller.go:141    reconciling gateways    {"runner": "provider"}
2024-04-07T08:29:53.884Z        INFO    provider        kubernetes/controller.go:719    processing Gateway      {"runner": "provider", "namespace": "default", "name": "eg"}
2024-04-07T08:29:53.884Z        INFO    provider        kubernetes/routes.go:235        processing HTTPRoute    {"runner": "provider", "namespace": "default", "name": "backend"}
2024-04-07T08:29:53.884Z        INFO    provider        kubernetes/controller.go:493    processing OIDC HMAC Secret     {"runner": "provider", "namespace": "envoy-gateway-system", "name": "envoy-oidc-hmac"}
2024-04-07T08:29:53.884Z        INFO    provider        kubernetes/controller.go:316    processing Backend      {"runner": "provider", "kind": "Service", "namespace": "default", "name": "backend"}
2024-04-07T08:29:53.884Z        INFO    provider        kubernetes/controller.go:330    added Service to resource tree  {"runner": "provider", "namespace": "default", "name": "backend"}
2024-04-07T08:29:53.884Z        INFO    provider        kubernetes/controller.go:364    added EndpointSlice to resource tree    {"runner": "provider", "namespace": "default", "name": "backend-nphpz"}
2024-04-07T08:29:53.884Z        INFO    provider        kubernetes/controller.go:277    reconciled gateways successfully        {"runner": "provider"}
2024-04-07T08:29:53.884Z        INFO    provider        status/status.go:122    received a status update        {"runner": "provider", "namespace": "", "name": "eg"}
2024-04-07T08:29:53.884Z        INFO    gateway-api     runner/runner.go:56     received an update      {"runner": "gateway-api"}
2024-04-07T08:29:53.884Z        INFO    provider.eg     status/status.go:92     status unchanged, bypassing update      {"runner": "provider"}
2024-04-07T08:29:53.884Z        INFO    gateway-api     runner/runner.go:104    proxy:
  listeners:
  - address: null
    name: default/eg/http
    ports:
    - containerPort: 8001
      name: http
      protocol: HTTP
      servicePort: 8001
  metadata:
    labels:
      gateway.envoyproxy.io/owning-gateway-name: eg
      gateway.envoyproxy.io/owning-gateway-namespace: default
  name: default/eg
        {"runner": "gateway-api", "infra-ir": "default/eg"}
2024-04-07T08:29:53.884Z        INFO    infrastructure  runner/runner.go:78     received an update      {"runner": "infrastructure"}
2024-04-07T08:29:53.884Z        INFO    gateway-api     runner/runner.go:115    accessLog:
  text:
  - path: /dev/stdout
http:
- address: 0.0.0.0
  hostnames:
  - '*'
  isHTTP2: false
  name: default/eg/http
  path:
    escapedSlashesAction: UnescapeAndRedirect
    mergeSlashes: true
  port: 8001
  routes:
  - backendWeights:
      invalid: 0
      valid: 0
    destination:
      name: httproute/default/backend/rule/0
      settings:
      - addressType: IP
        endpoints:
        - host: 192.168.49.6
          port: 3000
        protocol: HTTP
        weight: 1
    hostname: '*'
    isHTTP2: false
    name: httproute/default/backend/rule/0/match/0/*
    pathMatch:
      distinct: false
      name: ""
      prefix: /
        {"runner": "gateway-api", "xds-ir": "default/eg"}
2024-04-07T08:29:53.885Z        INFO    xds-translator  runner/runner.go:55     received an update      {"runner": "xds-translator"}
2024-04-07T08:29:53.885Z        INFO    provider        status/status.go:122    received a status update        {"runner": "provider", "namespace": "default", "name": "backend"}
2024-04-07T08:29:53.885Z        INFO    provider.backend.default        status/status.go:92     status unchanged, bypassing update      {"runner": "provider"}
2024-04-07T08:29:53.885Z        INFO    provider        status/status.go:122    received a status update        {"runner": "provider", "namespace": "default", "name": "eg"}
2024-04-07T08:29:53.885Z        INFO    xds-server      runner/runner.go:141    received an update      {"runner": "xds-server"}
2024-04-07T08:29:53.889Z        INFO    v3/simple.go:569        open delta watch ID:8 for type.googleapis.com/envoy.config.listener.v3.Listener Resources:map[] from nodeID: "envoy-default-eg-e41e7b31-97946df66-vt6wz",  version "5"
2024-04-07T08:29:53.910Z        ERROR   infrastructure  runner/runner.go:94     failed to create new infra      {"runner": "infrastructure", "error": "failed to create or update deployment envoy-gateway-system/envoy-default-eg-e41e7b31: for Update: Deployment.extensions \"envoy-default-eg-e41e7b31\" is invalid: spec.template.spec.containers[0].ports[1].name: Duplicate value: \"http-e0603c49\""}
2024-04-07T08:29:53.911Z        ERROR   watchable       message/watchutil.go:56 observed an error       {"runner": "infrastructure", "error": "failed to create or update deployment envoy-gateway-system/envoy-default-eg-e41e7b31: for Update: Deployment.extensions \"envoy-default-eg-e41e7b31\" is invalid: spec.template.spec.containers[0].ports[1].name: Duplicate value: \"http-e0603c49\""}

[zirain]

seems related to #3034, @ardikabs can you take a look?

[ardikabs]

as far as I can find is this kubernetes/kubernetes#113246, it turned out that the mergeKey refers to containerPort instead of name. Thought @zirain?

[ardikabs]

The solution would be moving Create to Patch as well, as per this comment

It is probably because you did not create the object using apply. Apply expects to manage the entire lifecycle of the object, so when you created independently, it had no record of the original applied content, so it didn't know your updated manifest should remove the old port.

Referred code:

gateway/internal/infrastructure/kubernetes/infra_client.go

Lines 32 to 35 in 29946b0

	// Create if it does not exist.
	if err := cli.Client.Create(ctx, specific); err != nil {
	return fmt.Errorf("for Create: %w", err)
	}

Just to let you know, I tested locally using Patch instead of Create on the above code, and it worked as intended (including changing the listener port).

[zirain]

can you send a fix and include a e2e test?