Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubevirt: Fix deploying kubevirt on git actions #1843

Merged
merged 1 commit into from
Aug 6, 2024

Conversation

oshoval
Copy link
Collaborator

@oshoval oshoval commented Aug 6, 2024

What this PR does / why we need it:
Since we are using git actions, the resources are limited.

We saw out of space errors, fix them by cleaning the container before
deploying the pods.

Also don't deploy some components to free more memory resources.

Special notes for your reviewer:
Fixes #1844

Release note:

None

@kubevirt-bot
Copy link
Collaborator

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@kubevirt-bot kubevirt-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Aug 6, 2024
@kubevirt-bot kubevirt-bot requested a review from phoracek August 6, 2024 06:30
@kubevirt-bot kubevirt-bot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Aug 6, 2024
@kubevirt-bot kubevirt-bot requested a review from qinqon August 6, 2024 06:30
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

/cc @RamLavi
lets try this approach as well
if it doesnt work, need to dump logs and such

@kubevirt-bot kubevirt-bot requested a review from RamLavi August 6, 2024 06:30
@kubevirt-bot kubevirt-bot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 6, 2024
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

nope didnt work

@oshoval oshoval changed the title kubevirt: Fix deploying kubevirt on git actions WIP kubevirt: Fix deploying kubevirt on git actions Aug 6, 2024
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

ok seems it is the wait bug, that need to be split
(there might be more changes needed, WIP)

hack/deploy-kubevirt.sh Outdated Show resolved Hide resolved
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.317.0/_diag/Worker_20240806-084825-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
System.IO.IOException: No space left on device : '/home/runner/runners/2.317.0/_diag/Worker_20240806-084825-utc.log'
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.Flush()
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing)
   at GitHub.Runner.Common.TraceManager.Dispose()
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing)
   at GitHub.Runner.Common.HostContext.Dispose()
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)
   at GitHub.Runner.Worker.Program.Main(String[] args)
System.IO.IOException: No space left on device : '/home/runner/runners/2.317.0/_diag/Worker_20240806-084825-utc.log'
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.Diagnostics.TextWriterTraceListener.Flush()
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id)
   at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message)
   at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
   at GitHub.Runner.Common.Tracing.Error(Exception exception)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)

i think we should move to prow (we have kubevirtci ready, drawback is that we must use old kind atm until we fix prow support for the new one)
this doesnt always happen fwiw, might and might not related to the original problem
but i do think that the real problem is about resources

i am still trying some stuff

@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

https://github.com/kubevirt/cluster-network-addons-operator/actions/runs/10263335275/job/28395542383?pr=1843
again storage problem, even that this PR changes are naive as far as i can tell
this is a new problem that started just on the last 2 runs

crypto/subtle: write /tmp/go-build334935846/b070/importcfg: no space left on device
crypto/internal/boring/sig: mkdir /tmp/go-build334935846/b072/: no space left on device

@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

Going to try to use kind-ovn kubevirtci provider that already worked on prow when was tested with kubevirt ipam PR
as the errors here arent recoverable easily or at all (limited resources on git actions for heavy testing requirement, as we deploy lots of pods)

due to the new blocking storage problem i could not even reach the state where i want to dump logs / try to deploy less components (which affected prow as well - on prow we could just add memory in this case)

@oshoval oshoval closed this Aug 6, 2024
@oshoval oshoval reopened this Aug 6, 2024
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

awesome passed now
https://github.com/kubevirt/cluster-network-addons-operator/actions/runs/10264945110/job/28400069861?pr=1843
thanks for the co-op Miguel

will rework the PR

@oshoval oshoval changed the title WIP kubevirt: Fix deploying kubevirt on git actions kubevirt: Fix deploying kubevirt on git actions Aug 6, 2024
@oshoval oshoval marked this pull request as ready for review August 6, 2024 11:13
@kubevirt-bot kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 6, 2024
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

ready for review please
thanks

Copy link
Contributor

@maiqueb maiqueb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need linux bridge plugin ?

automation/components-functests.setup.sh Show resolved Hide resolved
Since we are using git actions, the resources are limited.

We saw out of space errors, fix them by cleaning the container before
deploying the pods.

Also don't deploy some components to free more memory resources.

Signed-off-by: Or Shoval <[email protected]>
Copy link
Contributor

@maiqueb maiqueb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you.

@kubevirt-bot kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 6, 2024
Copy link

sonarqubecloud bot commented Aug 6, 2024

@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

/approve

@kubevirt-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: oshoval

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubevirt-bot kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 6, 2024
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

/test pull-e2e-cluster-network-addons-operator-monitoring-k8

@kubevirt-bot
Copy link
Collaborator

@oshoval: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test pull-cluster-network-addons-operator-unit-test
  • /test pull-e2e-cluster-network-addons-operator-br-marker-functests
  • /test pull-e2e-cluster-network-addons-operator-kubemacpool-functests
  • /test pull-e2e-cluster-network-addons-operator-lifecycle-k8s
  • /test pull-e2e-cluster-network-addons-operator-macvtap-cni-functests
  • /test pull-e2e-cluster-network-addons-operator-monitoring-k8s
  • /test pull-e2e-cluster-network-addons-operator-multus-functests
  • /test pull-e2e-cluster-network-addons-operator-ovs-cni-functests
  • /test pull-e2e-cluster-network-addons-operator-workflow-k8s
  • /test pull-e2e-cnao-kube-secondary-dns-functests
  • /test pull-e2e-cnao-multus-dynamic-networks-functests

Use /test all to run all jobs.

In response to this:

/test pull-e2e-cluster-network-addons-operator-monitoring-k8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

/test pull-e2e-cluster-network-addons-operator-monitoring-k8s

@oshoval
Copy link
Collaborator Author

oshoval commented Aug 6, 2024

@kubevirt-bot kubevirt-bot merged commit 52850e6 into kubevirt:main Aug 6, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. lgtm Indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. size/S
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[main] IPAM lane broken
3 participants