istio-ingressgateway-workload stuck in pending while testing in proxy

[misohu]

Bug Description

While testing in proxy environment the istio-ingressgateway-workload is stuck in pending. Metallb is enabled and its pods are running.

To Reproduce

1. Deploy Proxy environment on AWS Document the process of setting up a proxy environment to test CKF behind proxy bundle-kubeflow#990 (comment)
2. Setup microk8s and juju and deploy kubeflow https://charmed-kubeflow.io/docs/install-on-microk8s-behind-a-web-proxy#heading--prepare-your-environment
3. Run juju status isito-pilot will stuck in

istio-pilot/0*              error     idle   10.1.209.83                  hook failed: "ingress-relation-created"

Environment

Proxy environment
microk8s 1.29-strict
juju 3.4/stable
ckf 1.9/stabe

Relevant Log Output

unit-istio-pilot-0: 11:28:31 INFO unit.istio-pilot/0.juju-log ingress:12: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/kubeflow/services/istio-ingressgateway-workload "HTTP/1.1 200 OK"
unit-istio-pilot-0: 11:28:31 ERROR unit.istio-pilot/0.juju-log ingress:12: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1206, in <module>
    main(Operator)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/main.py", line 549, in main
    manager = _Manager(charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/main.py", line 432, in __init__
    self.charm = self._make_charm(self.framework, self.dispatcher)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/main.py", line 435, in _make_charm
    charm = self._charm_class(framework)
  File "./src/charm.py", line 116, in __init__
    cert_subject=self._cert_subject,
  File "./src/charm.py", line 477, in _cert_subject
    svc_address = _get_gateway_address_from_svc(svc)
  File "./src/charm.py", line 1060, in _get_gateway_address_from_svc
    gateway_address = _get_address_from_loadbalancer(svc)
  File "./src/charm.py", line 1075, in _get_address_from_loadbalancer
    if len(ingresses) != 1:
TypeError: object of type 'NoneType' has no len()
unit-istio-pilot-0: 11:28:31 ERROR juju.worker.uniter.operation hook "ingress-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
unit-istio-pilot-0: 11:28:31 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook
unit-istio-pilot-0: 11:30:58 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook
unit-istio-pilot-0: 11:33:31 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook
unit-istio-pilot-0: 11:33:32 INFO unit.istio-pilot/0.juju-log ingress:12: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/kubeflow/services/istio-ingressgateway-workload "HTTP/1.1 200 OK"
unit-istio-pilot-0: 11:33:32 ERROR unit.istio-pilot/0.juju-log ingress:12: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1206, in <module>
    main(Operator)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/main.py", line 549, in main
    manager = _Manager(charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/main.py", line 432, in __init__
    self.charm = self._make_charm(self.framework, self.dispatcher)
  File "/var/lib/juju/agents/unit-istio-pilot-0/charm/venv/ops/main.py", line 435, in _make_charm
    charm = self._charm_class(framework)
  File "./src/charm.py", line 116, in __init__
    cert_subject=self._cert_subject,
  File "./src/charm.py", line 477, in _cert_subject
    svc_address = _get_gateway_address_from_svc(svc)
  File "./src/charm.py", line 1060, in _get_gateway_address_from_svc
    gateway_address = _get_address_from_loadbalancer(svc)
  File "./src/charm.py", line 1075, in _get_address_from_loadbalancer
    if len(ingresses) != 1:
TypeError: object of type 'NoneType' has no len()
unit-istio-pilot-0: 11:33:32 ERROR juju.worker.uniter.operation hook "ingress-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
unit-istio-pilot-0: 11:33:32 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6138.

This message was autogenerated

[misohu]

Metallb was active

ubuntu@ip-10-0-134-52:~$ microk8s.kubectl get po -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-5f7bb57799-24lzz   1/1     Running   0          39m
speaker-nx9ht                 1/1     Running   0          34m
ubuntu@ip-10-0-134-52:~$ microk8s status
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    ingress              # (core) Ingress controller for external access
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    storage              # (core) Alias to hostpath-storage add-on, deprecated
  disabled:
    cert-manager         # (core) Cloud native certificate management
    cis-hardening        # (core) Apply CIS K8s hardening
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    host-access          # (core) Allow Pods connecting to Host services smoothly
    mayastor             # (core) OpenEBS MayaStor
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    rook-ceph            # (core) Distributed Ceph storage using Rook

[misohu]

Duplicate of this #287