kserve-integration test fails due to dependency conflicts for protobuf

[motjuste]

Bug Description

Solutions QA had a recent failure in running the Kubeflow UATs against CKF-1.9 where the test kserve-integration fails while installing dependencies using pip. We had a successful run earlier in the day, so this might be coming from some Python package getting updated in the meantime.

To Reproduce

1. Deploy Charmed-Kubernetes 1.30/stable using Juju 3.4/stable on AWS
2. Deploy Charmed-Kubeflow 1.9/stable
3. Clone, setup, and run the UATs as tox -e uats-remote -- --junitxml=/home/ubuntu/project/generated/kubeflow/charmed-kubeflow-uats/junit.xml --filter 'not mlflow'

Environment

• Juju 3.4/stable
• Charmed-Kubernetes 1.30/stable
• Charmed-Kubeflow 1.9/stable
• UATs from the main branch as of the date of the run.

Relevant Log Output

2025-01-22-18:53:08 root DEBUG     Found existing installation: google-api-core 2.23.0
2025-01-22-18:53:08 root DEBUG     Uninstalling google-api-core-2.23.0:
2025-01-22-18:53:08 root DEBUG       Successfully uninstalled google-api-core-2.23.0
2025-01-22-18:53:08 root DEBUG Successfully installed aiohttp-3.9.5 aiohttp-cors-0.7.0 aiosignal-1.3.1 annotated-types-0.7.0 cachetools-5.3.3 certifi-2024.7.4 charset-normalizer-3.3.2 cloudevents-1.11.0 colorful-0.5.6 deprecation-2.1.0 distlib-0.3.8 fastapi-0.109.2 filelock-3.15.4 frozenlist-1.4.1 google-api-core-2.19.1 google-auth-2.32.0 googleapis-common-protos-1.63.2 grpcio-1.51.3 httptools-0.6.1 httpx-0.26.0 idna-3.7 kserve-0.13.0 multidict-6.0.5 opencensus-0.11.4 opencensus-context-0.1.3 orjson-3.10.6 pandas-2.2.2 platformdirs-3.11.0 proto-plus-1.24.0 protobuf-3.20.3 psutil-5.9.8 py-spy-0.3.14 pyasn1-0.6.0 pyasn1-modules-0.4.0 pydantic-2.8.2 pydantic-core-2.20.1 python-dotenv-1.0.1 pyyaml-6.0.1 ray-2.10.0 smart-open-7.0.4 starlette-0.36.3 tenacity-8.5.0 timing-asgi-0.3.1 urllib3-2.2.2 uvicorn-0.21.1 uvloop-0.19.0 virtualenv-20.21.0 watchfiles-0.22.0 websockets-12.0 yarl-1.9.4
2025-01-22-18:53:08 root DEBUG ----------------------------- Captured stderr call -----------------------------
2025-01-22-18:53:08 root DEBUG ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2025-01-22-18:53:08 root DEBUG kfp-pipeline-spec 0.5.0 requires protobuf<5,>=4.21.1, but you have protobuf 3.20.3 which is incompatible.
2025-01-22-18:53:08 root DEBUG kubeflow-katib 0.17.0 requires protobuf<5,>=4.21.12, but you have protobuf 3.20.3 which is incompatible.
2025-01-22-18:53:08 root DEBUG kfp 2.10.1 requires protobuf<5,>=4.21.1, but you have protobuf 3.20.3 which is incompatible.
2025-01-22-18:53:08 root DEBUG kfp 2.10.1 requires urllib3<2.0.0, but you have urllib3 2.2.2 which is incompatible.
2025-01-22-18:53:08 root DEBUG ------------------------------ Captured log call -------------------------------
2025-01-22-18:53:08 root DEBUG INFO     test_notebooks:test_notebooks.py:48 Note that only CPU tests will be run. In order to run tests that use an NVIDIA GPU,use the `--include-gpu-tests` flag e.g. `tox -e kubeflow-local -- --include-gpu-tests`. To learn more, use `--help` or refer to the repository's README file.
2025-01-22-18:53:08 root DEBUG INFO     test_notebooks:test_notebooks.py:55 Running kserve-integration.ipynb...
2025-01-22-18:53:08 root DEBUG =========================== short test summary info ============================
2025-01-22-18:53:08 root DEBUG FAILED test_notebooks.py::test_notebook[kserve-integration] - Failed: Notebook execution failed with KeyError: 'status'
2025-01-22-18:53:08 root DEBUG ============ 1 failed, 4 passed, 4 deselected in 998.04s (0:16:38) ======
2025-01-22-18:53:08 root DEBUG =======

Additional Context

• Logs from the runner (may get deleted after a week): https://oil-jenkins.canonical.com/job/fce_validate/6391/consoleFull
  • More permanently available at: fce_validate_6391/fce_validate_6391_kubeflow_charmed-kubeflow-uats_console.out
• All logs and crashdumps: https://oil-jenkins.canonical.com/artifacts/a34e58dd-cccc-4fc6-8990-5bd670fbf25a/index.html
  • Relevant captured tox logs: generated/generated/kubeflow/charmed-kubeflow-uats/log/2-commands[0].log

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6781.

This message was autogenerated