Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rolling mlserver back from 1.3.5 to 1.3.2 #93

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Rolling mlserver back from 1.3.5 to 1.3.2 #93

wants to merge 7 commits into from

Conversation

BON4
Copy link

@BON4 BON4 commented Nov 19, 2024

Description

  • Edit rockcraft.yaml for 1.3.2 version
  • Run tests
  • Validate juju status

For Modified:

  • config.yaml, src/default-custom-images.json
  • tests/integration/test_data/expected_changed_seldon_cm.json
  • tests/integration/test_data/expected_seldon_cm.json

Replaced "seldonio/mlserver:1.3.5-alibi-explain" with "vyurchenko581/mlserver:1.3.2".

Tests PASSED. Test output:

charm-integration: 113 D clear env temp folder /home/bon/repos/seldon-core-operator/.tox/charm-integration/tmp [tox/tox_env/api.py:311]
charm-integration: 132 I find interpreter for spec PythonSpec(path=/usr/bin/python3) [virtualenv/discovery/builtin.py:58]
charm-integration: 132 I proposed PythonInfo(spec=CPython3.10.12.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:65]
charm-integration: 132 D accepted PythonInfo(spec=CPython3.10.12.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:67]
charm-integration: 133 D filesystem is case-sensitive [virtualenv/info.py:25]
charm-integration: 162 W commands[0]> pytest -vv --tb native --asyncio-mode=auto /home/bon/repos/seldon-core-operator/tests/integration/test_charm.py --log-cli-level=INFO -s --model kubeflow --keep-models [tox/tox_env/api.py:425]
============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.10.12, pytest-8.2.2, pluggy-1.5.0 -- /home/bon/repos/seldon-core-operator/.tox/charm-integration/bin/python
cachedir: .tox/charm-integration/.pytest_cache
rootdir: /home/bon/repos/seldon-core-operator
configfile: pyproject.toml
plugins: operator-0.35.0, anyio-4.4.0, asyncio-0.21.2
asyncio: mode=auto
collected 10 items                                                                                                                                                                                             

tests/integration/test_charm.py::test_build_and_deploy 
------------------------------------------------------------------------------------------------ live log setup ------------------------------------------------------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:734 Connecting to existing model microk8s-localhost:kubeflow on unspecified cloud
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:575 Using tmp_path: /home/bon/repos/seldon-core-operator/.tox/charm-integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:1083 Building charm seldon-core
INFO     pytest_operator.plugin:plugin.py:1088 Built charm seldon-core in 32.77s
INFO     juju.model:model.py:2097 Deploying local:seldon-core-0
INFO     juju.model:model.py:2971 Waiting for model:
  seldon-controller-manager/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:2971 Waiting for model:
  seldon-controller-manager/0 [allocating] waiting: agent initialising
INFO     juju.model:model.py:2971 Waiting for model:
  seldon-controller-manager/0 [allocating] waiting: agent initialising
INFO     juju.model:model.py:2971 Waiting for model:
  seldon-controller-manager/0 [executing] maintenance: Creating K8S resources
INFO     juju.model:model.py:2971 Waiting for model:
  seldon-controller-manager/0 [idle] active: 
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:78 deploying grafana-agent-k8s from latest/stable channel
INFO     juju.model:model.py:2097 Deploying ch:amd64/jammy/grafana-agent-k8s-45
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:82 Adding relation: seldon-controller-manager:grafana-dashboard and grafana-agent-k8s:grafana-dashboards-consumer
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:95 Adding relation: seldon-controller-manager:metrics-endpoint and grafana-agent-k8s:metrics-endpoint
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:108 Adding relation: seldon-controller-manager:logging and grafana-agent-k8s:logging-provider
INFO     juju.model:model.py:2971 Waiting for model:
  grafana-agent-k8s/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:2971 Waiting for model:
  grafana-agent-k8s/0 [allocating] waiting: agent initialising
INFO     juju.model:model.py:2971 Waiting for model:
  grafana-agent-k8s/0 [executing] maintenance: installing charm software
INFO     juju.model:model.py:2971 Waiting for model:
  grafana-agent-k8s/0 [idle] blocked: grafana-cloud-config: off, grafana-dashboards-provider: off
INFO     juju.model:model.py:2971 Waiting for model:
  grafana-agent-k8s/0 [idle] blocked: grafana-cloud-config: off, grafana-dashboards-provider: off
PASSED
tests/integration/test_charm.py::test_alert_rules 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     test_charm:test_charm.py:92 found alert_rules: {'SeldonUnfinishedWorkIncrease', 'SeldonWebhookError', 'SeldonHTTPError', 'KubeflowServiceDown', 'SeldonReconcileError', 'SeldonWorkqueueTooManyRetries', 'KubeflowServiceIsNotStable'}
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:225 found relations [<Relation id=2 grafana-agent-k8s:metrics-endpoint seldon-controller-manager:metrics-endpoint>] for seldon-controller-manager:metrics-endpoint
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:371 running cmd `relation-get --format=yaml -r 2 --app - seldon-controller-manager` on unit seldon-controller-manager/0
PASSED
tests/integration/test_charm.py::test_metrics_endpoints 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:225 found relations [<Relation id=2 grafana-agent-k8s:metrics-endpoint seldon-controller-manager:metrics-endpoint>] for seldon-controller-manager:metrics-endpoint
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:371 running cmd `relation-get --format=yaml -r 2 --app - seldon-controller-manager` on unit seldon-controller-manager/0
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:371 running cmd `curl -m 5 -sS localhost:12345/agent/api/v1/metrics/targets` on unit grafana-agent-k8s/0
PASSED
tests/integration/test_charm.py::test_logging 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:225 found relations [<Relation id=3 seldon-controller-manager:logging grafana-agent-k8s:logging-provider>] for seldon-controller-manager:logging
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:371 running cmd `relation-get --format=yaml -r 3 - grafana-agent-k8s/0` on unit grafana-agent-k8s/0
PASSED
tests/integration/test_charm.py::test_grafana_dashboards 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     test_charm:test_charm.py:112 found dashboards: {'seldon-core.json'}
PASSED
tests/integration/test_charm.py::test_configmap_created 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/api/v1/namespaces/kubeflow/configmaps/seldon-config "HTTP/1.1 200 OK"
PASSED
tests/integration/test_charm.py::test_configmap_changes_with_config 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     juju.model:model.py:2971 Waiting for model:
  seldon-controller-manager/0 [idle] active: 
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/api/v1/namespaces/kubeflow/configmaps/seldon-config "HTTP/1.1 200 OK"
INFO     juju.model:model.py:2971 Waiting for model:
  seldon-controller-manager/0 [idle] active: 
PASSED
tests/integration/test_charm.py::test_seldon_istio_relation 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     juju.model:model.py:2097 Deploying ch:amd64/focal/istio-gateway-1287
INFO     juju.model:model.py:2097 Deploying ch:amd64/focal/istio-pilot-1235
INFO     juju.model:model.py:2971 Waiting for model:
  istio-pilot/0 [allocating] waiting: installing agent
  istio-ingressgateway/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:2971 Waiting for model:
  istio-pilot/0 [allocating] waiting: agent initialising
  istio-ingressgateway/0 [executing] waiting: List of <ops.model.Relation istio-pilot:5> versions not found for apps: istio-pilot
INFO     juju.model:model.py:2971 Waiting for model:
  istio-pilot/0 [executing] maintenance: installing charm software
  istio-ingressgateway/0 [idle] waiting: List of <ops.model.Relation istio-pilot:5> versions not found for apps: istio-pilot
INFO     juju.model:model.py:2971 Waiting for model:
  istio-pilot/0 [executing] active: 
  istio-ingressgateway/0 [executing] waiting: List of <ops.model.Relation istio-pilot:5> versions not found for apps: istio-pilot
INFO     juju.model:model.py:2971 Waiting for model:
  istio-pilot/0 [idle] active: 
  istio-ingressgateway/0 [idle] active: 
  seldon-controller-manager/0 [idle] active: 
INFO     juju.model:model.py:2971 Waiting for model:
  seldon-controller-manager/0 [idle] active: 
PASSED
tests/integration/test_charm.py::test_seldon_deployment 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/api/v1/namespaces/default "HTTP/1.1 200 OK"
INFO     httpx:_client.py:1026 HTTP Request: PATCH https://127.0.0.1:16443/api/v1/namespaces/default "HTTP/1.1 200 OK"
INFO     httpx:_client.py:1026 HTTP Request: POST https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments "HTTP/1.1 201 Created"
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == None (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:29 seldondeployment/seldon-model status == Creating (waiting for 'Available')
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:22 seldondeployment/seldon-model status == Available
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/api/v1/namespaces/default/services/seldon-model-example-classifier "HTTP/1.1 200 OK"
INFO     httpx:_client.py:1026 HTTP Request: DELETE https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model?gracePeriodSeconds=0 "HTTP/1.1 200 OK"
INFO     test_charm:utils.py:43 Waiting for <class 'lightkube.generic_resource.seldondeployment'>/seldon-model-1 to be deleted.
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/seldon-model-1 "HTTP/1.1 404 Not Found"
INFO     test_charm:utils.py:48 Not found <class 'lightkube.generic_resource.seldondeployment'>/seldon-model-1. Status 404 
INFO     juju.model:model.py:2971 Waiting for model:
  seldon-controller-manager/0 [idle] active: 
INFO     juju.model:model.py:2971 Waiting for model:
  seldon-controller-manager/0 [idle] active: 
PASSED
tests/integration/test_charm.py::test_remove_with_resources_present 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     test_charm:utils.py:43 Waiting for <class 'lightkube.resources.core_v1.Pod'>/seldon-controller-manager-0 to be deleted.
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/api/v1/namespaces/kubeflow/pods/seldon-controller-manager-0 "HTTP/1.1 404 Not Found"
INFO     test_charm:utils.py:48 Not found <class 'lightkube.resources.core_v1.Pod'>/seldon-controller-manager-0. Status 404 
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.juju.is%2Fcreated-by%3Dseldon-controller-manager "HTTP/1.1 200 OK"
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/api/v1/namespaces/kubeflow/configmaps/seldon-config "HTTP/1.1 404 Not Found"
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/api/v1/namespaces/kubeflow/configmaps/a33bd623.machinelearning.seldon.io "HTTP/1.1 404 Not Found"
INFO     httpx:_client.py:1026 HTTP Request: GET https://127.0.0.1:16443/api/v1/namespaces/kubeflow/services?labelSelector=app.juju.is%2Fcreated-by%3Dseldon-controller-manager "HTTP/1.1 200 OK"
PASSED
---------------------------------------------------------------------------------------------- live log teardown -----------------------------------------------------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:862 Model status:

Model     Controller          Cloud/Region        Version  SLA          Timestamp
kubeflow  microk8s-localhost  microk8s/localhost  3.4.6    unsupported  15:23:35+02:00

App                   Version  Status   Scale  Charm              Channel         Rev  Address         Exposed  Message
grafana-agent-k8s     0.32.1   waiting      1  grafana-agent-k8s  latest/stable    45  10.152.183.96   no       installing agent
istio-ingressgateway           active       1  istio-gateway      latest/edge    1287  10.152.183.67   no       
istio-pilot                    active       1  istio-pilot        latest/edge    1235  10.152.183.119  no       

Unit                     Workload  Agent  Address      Ports  Message
grafana-agent-k8s/0*     blocked   idle   10.1.34.143         send-remote-write: off, grafana-cloud-config: off
istio-ingressgateway/0*  active    idle   10.1.34.129         
istio-pilot/0*           active    idle   10.1.34.130         

INFO     pytest_operator.plugin:plugin.py:868 Juju error logs:


INFO     pytest_operator.plugin:plugin.py:947 Forgetting model main...


======================================================================================== 10 passed in 824.86s (0:13:44) ========================================================================================
charm-integration: 825816 I exit 0 (825.65 seconds) /home/bon/repos/seldon-core-operator> pytest -vv --tb native --asyncio-mode=auto /home/bon/repos/seldon-core-operator/tests/integration/test_charm.py --log-cli-level=INFO -s --model kubeflow --keep-models pid=201502 [tox/execute/api.py:280]
  charm-integration: OK (825.70=setup[0.05]+cmd[825.65] seconds)
  congratulations :) (825.75 seconds)

juju status:

Model     Controller          Cloud/Region        Version  SLA          Timestamp
kubeflow  microk8s-localhost  microk8s/localhost  3.4.6    unsupported  15:27:05+02:00

App                   Version  Status   Scale  Charm              Channel         Rev  Address         Exposed  Message
grafana-agent-k8s     0.32.1   waiting      1  grafana-agent-k8s  latest/stable    45  10.152.183.96   no       installing agent
istio-ingressgateway           active       1  istio-gateway      latest/edge    1287  10.152.183.67   no       
istio-pilot                    active       1  istio-pilot        latest/edge    1235  10.152.183.119  no       

Unit                     Workload  Agent  Address      Ports  Message
grafana-agent-k8s/0*     blocked   idle   10.1.34.143         send-remote-write: off, grafana-cloud-config: off
istio-ingressgateway/0*  active    idle   10.1.34.129         
istio-pilot/0*           active    idle   10.1.34.130

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@BON4