Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(cherry-pick to track/1.12) fix: configure proxy env vars in serving controller (#208) #218

Merged

Conversation

NohaIhab
Copy link
Contributor

@NohaIhab NohaIhab commented Aug 19, 2024

backports #208 to track/1.12

Testing the upgrade

  1. Deploy the following bundle:
bundle: kubernetes
name: kubeflow
docs: https://discourse.charmhub.io/t/3749
applications:
  istio-ingressgateway:
    charm: istio-gateway
    channel: 1.22/stable
    scale: 1
    trust: true
    _github_repo_name: istio-operators
    _github_repo_branch: main
    options:
      kind: ingress
  istio-pilot:
    charm: istio-pilot
    channel: 1.22/stable
    scale: 1
    trust: true
    _github_repo_name: istio-operators
    _github_repo_branch: main
    options:
      default-gateway: kubeflow-gateway
  knative-operator:
    charm: knative-operator
    channel: 1.12/stable
    scale: 1
    trust: true
    _github_repo_name: knative-operators
    _github_repo_branch: main
  knative-serving:
    charm: knative-serving
    channel: 1.12/stable
    scale: 1
    trust: true
    options:
      namespace: knative-serving
      istio.gateway.namespace: kubeflow
      istio.gateway.name: kubeflow-gateway
    _github_repo_name: knative-operators
    _github_repo_branch: main
  kserve-controller:
    charm: kserve-controller
    channel: 0.13/stable
    scale: 1
    trust: true
    options:
      deployment-mode: serverless
    _github_repo_name: kserve-operators
    _github_repo_branch: main
relations:
  - [istio-pilot:istio-pilot, istio-ingressgateway:istio-pilot]
  - [kserve-controller:ingress-gateway, istio-pilot:gateway-info]
  - [kserve-controller:local-gateway, knative-serving:local-gateway]
  1. Create a namespace to test with
kubectl create namespace testing
  1. Apply KSVC example cloudevents-player to testing namespace:
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: cloudevents-player
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: "1"
    spec:
      containers:
        - image: ruromero/cloudevents-player:latest
          env:
            - name: BROKER_URL
              value: http://broker-ingress.knative-eventing.svc.cluster.local/default/example-broker
  1. Check KSVC is Ready
kubectl get ksvc -n testing
NAME                 URL                                                     LATESTCREATED              LATESTREADY                READY   REASON
cloudevents-player   http://cloudevents-player.testing.10.64.140.43.nip.io   cloudevents-player-00001   cloudevents-player-00001   True    
  1. Refresh knative-serving to the charm from this PR
juju refresh knative-serving --channel=1.12/edge/pr-218

After the upgrade

  • charm went to active after refresh
juju status
Model     Controller          Cloud/Region        Version  SLA          Timestamp
kubeflow  microk8s-localhost  microk8s/localhost  3.4.5    unsupported  10:29:36Z

App                   Version  Status  Scale  Charm              Channel            Rev  Address         Exposed  Message
istio-ingressgateway           active      1  istio-gateway      1.22/stable       1127  10.152.183.176  no       
istio-pilot                    active      1  istio-pilot        1.22/stable       1079  10.152.183.49   no       
knative-operator               active      1  knative-operator   1.12/stable        433  10.152.183.224  no       
knative-serving                active      1  knative-serving    1.12/edge/pr-218   480  10.152.183.141  no       
kserve-controller              active      1  kserve-controller  0.13/stable        597  10.152.183.135  no       

Unit                     Workload  Agent  Address       Ports  Message
istio-ingressgateway/0*  active    idle   10.1.199.206         
istio-pilot/0*           active    idle   10.1.199.207         
knative-operator/0*      active    idle   10.1.199.208         
knative-serving/0*       active    idle   10.1.199.222         
kserve-controller/0*     active    idle   10.1.199.210         
  • ksvc created before upgrade is unaffected
kubectl get ksvc -n testing

kubectl get ksvc -n testing
NAME                 URL                                                     LATESTCREATED              LATESTREADY                READY   REASON
cloudevents-player   http://cloudevents-player.testing.10.64.140.43.nip.io   cloudevents-player-00001   cloudevents-player-00001   True    
  • check the new charm configs for proxy exist
juju config knative-serving

juju config knative-serving
application: knative-serving
application-config: 
...
charm: knative-serving
settings: 
...
  http-proxy: 
    default: ""
    description: The value of HTTP_PROXY environment variable in the serving controller.
    source: default
    type: string
    value: ""
  https-proxy: 
    default: ""
    description: The value of HTTPS_PROXY environment variable in the serving controller.
    source: default
    type: string
    value: ""
...
  no-proxy: 
    default: ""
    description: The value of NO_PROXY environment variable in the serving controller.
    source: default
    type: string
    value: ""
...
  • Set the new proxy configs and check it takes effect in KnativeServing CR
juju config knative-serving http-proxy="test-http-proxy" https-proxy="test-https-proxy" no-proxy="test-no-proxy"

Get KnativeServing:

kubectl get KnativeServing -n knative-serving knative-serving -oyaml

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  creationTimestamp: "2024-08-21T10:19:28Z"
  finalizers:
  - knativeservings.operator.knative.dev
  generation: 2
  name: knative-serving
  namespace: knative-serving
  resourceVersion: "9101"
  uid: 3f8a7ab9-a465-43be-8c96-436bba12fd43
spec:
  config:
    deployment:
      progress-deadline: 600s
      registries-skipping-tag-resolving: nvcr.io
    domain:
      10.64.140.43.nip.io: ""
    features:
      kubernetes.podspec-affinity: enabled
      kubernetes.podspec-nodeselector: enabled
      kubernetes.podspec-tolerations: enabled
    istio:
      gateway.kubeflow.kubeflow-gateway: some-workload.knative-operator.svc.cluster.local
      local-gateway.knative-serving.knative-local-gateway: knative-local-gateway.kubeflow.svc.cluster.local
  version: 1.12.4
  workloads:
  - env:
    - container: controller
      envVars:
      - name: HTTP_PROXY
        value: test-http-proxy
      - name: HTTPS_PROXY
        value: test-https-proxy
      - name: NO_PROXY
        value: test-no-proxy
    name: controller
status:
  conditions:
  - lastTransitionTime: "2024-08-21T10:19:44Z"
    status: "True"
    type: DependenciesInstalled
  - lastTransitionTime: "2024-08-21T10:20:02Z"
    status: "True"
    type: DeploymentsAvailable
  - lastTransitionTime: "2024-08-21T10:19:44Z"
    status: "True"
    type: InstallSucceeded
  - lastTransitionTime: "2024-08-21T10:20:02Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-08-21T10:19:28Z"
    status: "True"
    type: VersionMigrationEligible
  manifests:
  - /var/run/ko/knative-serving/1.12.4
  - /var/run/ko/ingress/1.12/istio
  observedGeneration: 1
  version: 1.12.4

* feat: add initial config for proxy envs in knative-serving controller
* unit test proxy configs
* add integration test
@NohaIhab NohaIhab requested a review from a team as a code owner August 19, 2024 11:41
Copy link
Contributor

@mvlassis mvlassis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@NohaIhab NohaIhab merged commit e4f3b85 into track/1.12 Aug 21, 2024
14 checks passed
@NohaIhab NohaIhab deleted the cherrypick-fix-configure-proxy-env-vars-in-serving-1.12 branch August 21, 2024 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants