Running the UATs on EKS deployment fails with "requirements.txt file not found, pytest not found" #55

orfeas-k · 2023-11-22T16:38:47Z

Bug Description

Running the UATs using the driver for an EKS deployment fails with

ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
bash: line 2: pytest: command not found

Note that running them inside a notebook through the UI works.

To Reproduce

Deploy CKF to EKS following this guide and then try to run the UATs from your local terminal, as instructed in the README.md.

Environment

EKS 1.25
Juju 3.1
CKF 1.8/stable

Relevant Log Output

╰─$ tox -c ~/github/charmed-kubeflow-uats/ -e kubeflow                                                                                                                                                                             1 ↵
kubeflow: recreate env because python changed virtualenv version='20.24.7'->'20.24.5'
kubeflow: remove tox env folder /home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow
kubeflow: install_deps> python -I -m pip install -r requirements.txt
kubeflow: commands[0]> pytest -vv --tb native /home/orfeas/github/charmed-kubeflow-uats/driver/ -s --filter 'not mlflow' --model kubeflow
========================================================================================================= test session starts =========================================================================================================
platform linux -- Python 3.8.18, pytest-7.4.2, pluggy-1.3.0 -- /home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/bin/python
cachedir: .tox/kubeflow/.pytest_cache
rootdir: /home/orfeas/github/charmed-kubeflow-uats
configfile: pyproject.toml
plugins: anyio-4.0.0, asyncio-0.21.1, operator-0.29.0
asyncio: mode=strict
collected 2 items                                                                                                                                                                                                                     

driver/test_kubeflow_workloads.py::test_create_profile 
----------------------------------------------------------------------------------------------------------- live log setup ------------------------------------------------------------------------------------------------------------
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:59 Creating Profile test-kubeflow...
INFO     httpx:_client.py:1013 HTTP Request: POST https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/kubeflow.org/v1/profiles "HTTP/1.1 201 Created"
------------------------------------------------------------------------------------------------------------ live log call ------------------------------------------------------------------------------------------------------------
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 200 OK"
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/api/v1/namespaces/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:32 Waiting for namespace test-kubeflow to become 'Active': phase == Active
PASSED
driver/test_kubeflow_workloads.py::test_kubeflow_workloads 
------------------------------------------------------------------------------------------------------------ live log call ------------------------------------------------------------------------------------------------------------
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:100 Starting Kubernetes Job test-kubeflow/test-kubeflow to run notebook tests...
INFO     httpx:_client.py:1013 HTTP Request: POST https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs "HTTP/1.1 201 Created"
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == not ready)
INFO     utils:utils.py:40 Retrying in 2 seconds (attempts: 1)
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == not ready)
INFO     utils:utils.py:40 Retrying in 4 seconds (attempts: 2)
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == not ready)
INFO     utils:utils.py:40 Retrying in 8 seconds (attempts: 3)
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:40 Retrying in 16 seconds (attempts: 4)
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:40 Retrying in 32 seconds (attempts: 5)
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:123 Fetching Job logs...
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
bash: line 2: pytest: command not found
FAILED
---------------------------------------------------------------------------------------------------------- live log teardown ----------------------------------------------------------------------------------------------------------
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:72 Deleting Profile test-kubeflow...
INFO     httpx:_client.py:1013 HTTP Request: DELETE https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:129 Deleting Job test-kubeflow/test-kubeflow...
INFO     httpx:_client.py:1013 HTTP Request: DELETE https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"


============================================================================================================== FAILURES ===============================================================================================================
_______________________________________________________________________________________________________ test_kubeflow_workloads _______________________________________________________________________________________________________
Traceback (most recent call last):
  File "/home/orfeas/github/charmed-kubeflow-uats/driver/test_kubeflow_workloads.py", line 116, in test_kubeflow_workloads
    wait_for_job(lightkube_client, JOB_NAME, NAMESPACE)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/tenacity/__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/tenacity/__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/tenacity/__init__.py", line 314, in iter
    return fut.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "/home/orfeas/github/charmed-kubeflow-uats/driver/utils.py", line 72, in wait_for_job
    raise ValueError(f"Job {namespace}/{job_name} failed!")
ValueError: Job test-kubeflow/test-kubeflow failed!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/orfeas/github/charmed-kubeflow-uats/driver/test_kubeflow_workloads.py", line 118, in test_kubeflow_workloads
    pytest.fail(
  File "/home/orfeas/github/charmed-kubeflow-uats/.tox/kubeflow/lib/python3.8/site-packages/_pytest/outcomes.py", line 198, in fail
    raise Failed(msg=reason, pytrace=pytrace)
Failed: Something went wrong while running Job test-kubeflow/test-kubeflow. Please inspect the attached logs for more info...
---------------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------------
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:100 Starting Kubernetes Job test-kubeflow/test-kubeflow to run notebook tests...
INFO     httpx:_client.py:1013 HTTP Request: POST https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs "HTTP/1.1 201 Created"
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == not ready)
INFO     utils:utils.py:40 Retrying in 2 seconds (attempts: 1)
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == not ready)
INFO     utils:utils.py:40 Retrying in 4 seconds (attempts: 2)
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == not ready)
INFO     utils:utils.py:40 Retrying in 8 seconds (attempts: 3)
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:40 Retrying in 16 seconds (attempts: 4)
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     utils:utils.py:76 Waiting for Job test-kubeflow/test-kubeflow to complete (status == active)
INFO     utils:utils.py:40 Retrying in 32 seconds (attempts: 5)
INFO     httpx:_client.py:1013 HTTP Request: GET https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:123 Fetching Job logs...
-------------------------------------------------------------------------------------------------------- Captured log teardown --------------------------------------------------------------------------------------------------------
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:72 Deleting Profile test-kubeflow...
INFO     httpx:_client.py:1013 HTTP Request: DELETE https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/kubeflow.org/v1/profiles/test-kubeflow "HTTP/1.1 200 OK"
INFO     test_kubeflow_workloads:test_kubeflow_workloads.py:129 Deleting Job test-kubeflow/test-kubeflow...
INFO     httpx:_client.py:1013 HTTP Request: DELETE https://6a4ac0e0615768a2261c6b8a99a07eb7.yl4.eu-west-3.eks.amazonaws.com/apis/batch/v1/namespaces/test-kubeflow/jobs/test-kubeflow "HTTP/1.1 200 OK"
======================================================================================================= short test summary info =======================================================================================================
FAILED driver/test_kubeflow_workloads.py::test_kubeflow_workloads - Failed: Something went wrong while running Job test-kubeflow/test-kubeflow. Please inspect the attached logs for more info...
=============================================================================================== 1 failed, 1 passed in 67.31s (0:01:07) ================================================================================================
kubeflow: exit 1 (68.05 seconds) /home/orfeas/github/charmed-kubeflow-uats> pytest -vv --tb native /home/orfeas/github/charmed-kubeflow-uats/driver/ -s --filter 'not mlflow' --model kubeflow pid=1196807
  kubeflow: FAIL code 1 (82.06=setup[14.01]+cmd[68.05] seconds)
  evaluation failed :( (82.22 seconds)

Additional Context

No response

The text was updated successfully, but these errors were encountered:

orfeas-k · 2024-03-27T08:53:31Z

This is probably a duplicate of #65. However, we need to confirm that the solution for it will resolve the issue for EKS too.

syncronize-issues-to-jira · 2024-03-27T08:53:40Z

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5491.

This message was autogenerated

orfeas-k added the bug Something isn't working label Nov 22, 2023

orfeas-k changed the title ~~Running the UATs with EKS fails with "requirements.txt file not found, pytest not found"~~ Running the UATs on EKS deployment fails with "requirements.txt file not found, pytest not found" Nov 22, 2023

orfeas-k mentioned this issue Mar 27, 2024

feat: Run UATs from a remote commit #67

Merged

orfeas-k mentioned this issue May 28, 2024

Run UATs on EKS for v1.8 canonical/bundle-kubeflow#906

Merged

orfeas-k closed this as completed May 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running the UATs on EKS deployment fails with "requirements.txt file not found, pytest not found" #55

Running the UATs on EKS deployment fails with "requirements.txt file not found, pytest not found" #55

orfeas-k commented Nov 22, 2023 •

edited

Loading

orfeas-k commented Mar 27, 2024

syncronize-issues-to-jira bot commented Mar 27, 2024

Running the UATs on EKS deployment fails with "requirements.txt file not found, pytest not found" #55

Running the UATs on EKS deployment fails with "requirements.txt file not found, pytest not found" #55

Comments

orfeas-k commented Nov 22, 2023 • edited Loading

Bug Description

To Reproduce

Environment

Relevant Log Output

Additional Context

orfeas-k commented Mar 27, 2024

syncronize-issues-to-jira bot commented Mar 27, 2024

orfeas-k commented Nov 22, 2023 •

edited

Loading