Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pytorch plugin e2e probabilistic failure #2777

Closed
wangyang0616 opened this issue Apr 12, 2023 · 1 comment · Fixed by #2780
Closed

Pytorch plugin e2e probabilistic failure #2777

wangyang0616 opened this issue Apr 12, 2023 · 1 comment · Fixed by #2780
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@wangyang0616
Copy link
Member

What happened:
The pytorch-plugin e2e fails probabilistically.

During the preliminary analysis, sometimes the mirror image: volcano/pytorch-mnist-v1beta1-9ee8fda-example:0.0.1 fails to download, causing the use case to fail.

The error message is as follows:

Will run 32 of 32 specs
••••
------------------------------
• [FAILED] [675.825 seconds]
Pytorch Plugin E2E Test [It] will run and complete finally
/home/runner/work/volcano/volcano/test/e2e/jobseq/pytorch_plugin.go:12

  Timeline >>
  STEP: Initializing test context @ 04/11/23 12:51:24.026
  STEP: Creating Queues @ 04/11/23 12:51:24.228
  STEP: Cleaning up test context @ 04/11/23 13:02:22.847
  [FAILED] in [It] - /home/runner/work/volcano/volcano/test/e2e/jobseq/pytorch_plugin.go:56 @ 04/11/23 13:02:39.85
  << Timeline

  [FAILED] Unexpected error:
      <*errors.errorString | 0xc00001c120>: {
          s: "[Wait time out]: expected job 'pytorch-job' to be in status Completed, actual get Running",
      }
      [Wait time out]: expected job 'pytorch-job' to be in status Completed, actual get Running
  occurred
  In [It] at: /home/runner/work/volcano/volcano/test/e2e/jobseq/pytorch_plugin.go:56 @ 04/11/23 13:02:39.85
------------------------------
•••••••••••••••••••••••••••

Summarizing 1 Failure:
  [FAIL] Pytorch Plugin E2E Test [It] will run and complete finally
  /home/runner/work/volcano/volcano/test/e2e/jobseq/pytorch_plugin.go:56

Ran 32 of 32 Specs in 1693.664 seconds
FAIL! -- 31 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestE2E (1693.67s)
FAIL

Ginkgo ran 1 suite in 30m14.639645[387](https://github.com/volcano-sh/volcano/actions/runs/4667599932/jobs/8263602086?pr=2775#step:7:388)s

What you expected to happen:
CI use case runs run successfully every time.

How to reproduce it (as minimally and precisely as possible):
make e2e

Anything else we need to know?:

Environment:

  • Volcano Version: master
  • Kubernetes version (use kubectl version): k8s v1.25
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
@wangyang0616 wangyang0616 added the kind/bug Categorizes issue or PR as related to a bug. label Apr 12, 2023
@wangyang0616
Copy link
Member Author

/assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant