kubevirt,presubmits: update storage lanes to run with etcd in mem

[brianmcarey]

What this PR does / why we need it:

The storage e2e lanes see a large number of lane failures due to etcd timeouts[1] - running these lanes with etcd in memory should stop these etcd timeouts from occurring as we have seen with the other e2e lanes.

We saw instability when running with the default tmpfs size of 512M as this was filling up and causing etcd to restart.

{"level":"fatal","ts":"2024-09-18T12:16:27.580544Z","caller":"etcdserver/raft.go:224","msg":"failed to save Raft hard state and entries","error":"no space left on device","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*raftNode).start.func1\n\tgo.etcd.io/etcd/server/v3/etcdserver/raft.go:224"}

Increasing this to 1G improves the stability greatly.

[1] https://search.ci.kubevirt.io/?search=etcdserver%3A+request+timed+out&maxAge=336h&context=1&type=build-log&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

Release note:

[kubevirt-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[brianmcarey]

/rehearse

[kubevirt-bot]

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

[brianmcarey]

/rehearse

[kubevirt-bot]

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

[brianmcarey]

/rehearse

[kubevirt-bot]

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

[brianmcarey]

/rehearse

[kubevirt-bot]

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

[brianmcarey]

/rehearse

[kubevirt-bot]

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.31-sig-storage

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

[akalenyu]

@brianmcarey how is this looking? I think we're open minded about trying in mem etcd for a while to alleviate failures
@mhenriks wdyt

[brianmcarey]

@brianmcarey how is this looking? I think we're open minded about trying in mem etcd for a while to alleviate failures @mhenriks wdyt

Not 100% sure on the stability of this - I want to run a few more rehearsals to see first. I checked the memory usage and it looks ok for the test pod with this enabled.

[akalenyu]

@brianmcarey how is this looking? I think we're open minded about trying in mem etcd for a while to alleviate failures @mhenriks wdyt

Not 100% sure on the stability of this - I want to run a few more rehearsals to see first. I checked the memory usage and it looks ok for the test pod with this enabled.

Correct me if I'm wrong but the test job definitions are also uncapped on the memory size (only request is specified)

[brianmcarey]

@brianmcarey how is this looking? I think we're open minded about trying in mem etcd for a while to alleviate failures @mhenriks wdyt

Not 100% sure on the stability of this - I want to run a few more rehearsals to see first. I checked the memory usage and it looks ok for the test pod with this enabled.

Correct me if I'm wrong but the test job definitions are also uncapped on the memory size (only request is specified)

Yes they do not have any memory limits set.

[brianmcarey]

/rehearse

[kubevirt-bot]

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.31-sig-storage

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

[brianmcarey]

/rehearse

[kubevirt-bot]

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-k8s-1.31-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

[brianmcarey]

/rehearse

[kubevirt-bot]

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.31-sig-storage

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

[brianmcarey]

/rehearse

[kubevirt-bot]

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.31-sig-storage

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

[akalenyu]

Would it make sense to also apply this on 1.3/1.2 lanes?

[brianmcarey]

/rehearse

[kubevirt-bot]

Rehearsal jobs created for this PR:

rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage-1.3
rehearsal-pull-kubevirt-e2e-k8s-1.28-sig-storage-1.3
rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage-1.3
rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.31-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.30-sig-storage
rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage-1.2
rehearsal-pull-kubevirt-e2e-k8s-1.28-sig-storage-1.2
rehearsal-pull-kubevirt-e2e-k8s-1.27-sig-storage-1.2

You can trigger rehearsal for all jobs by commenting either /rehearse or /rehearse all
on this PR.

For a specific PR you can comment /rehearse {job-name}.

For a list of jobs that you can rehearse you can comment /rehearse ?.

[dhiller]

/approve

Thank you @brianmcarey !

Release

/hold

after the rehearsals are good.

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dhiller

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/OWNERS [dhiller]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[akalenyu]

/lgtm

[xpivarc]

@dhiller I think we are good to unhold

[akalenyu]

@brianmcarey: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
rehearsal-pull-kubevirt-e2e-k8s-1.29-sig-storage-1.2 ead44d6 link unknown /test pull-kubevirt-e2e-k8s-1.29-sig-storage-1.2
rehearsal-pull-kubevirt-e2e-k8s-1.28-sig-storage-1.2 ead44d6 link unknown /test pull-kubevirt-e2e-k8s-1.28-sig-storage-1.2

I don't think the 1.2 lanes are failing because of anything etcd or storage related. It looks like an error occurs in guest-console-log

[brianmcarey]

/hold cancel

Rehearsals look good.

[kubevirt-bot]

@brianmcarey: Updated the job-config configmap in namespace kubevirt-prow at cluster default using the following files:

• key kubevirt-presubmits-1.2.yaml using file github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-presubmits-1.2.yaml
• key kubevirt-presubmits-1.3.yaml using file github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-presubmits-1.3.yaml
• key kubevirt-presubmits.yaml using file github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-presubmits.yaml

In response to this:

What this PR does / why we need it:

The storage e2e lanes see a large number of lane failures due to etcd timeouts[1] - running these lanes with etcd in memory should stop these etcd timeouts from occurring as we have seen with the other e2e lanes.

We saw instability when running with the default tmpfs size of 512M as this was filling up and causing etcd to restart.

{"level":"fatal","ts":"2024-09-18T12:16:27.580544Z","caller":"etcdserver/raft.go:224","msg":"failed to save Raft hard state and entries","error":"no space left on device","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*raftNode).start.func1\n\tgo.etcd.io/etcd/server/v3/etcdserver/raft.go:224"}

Increasing this to 1G improves the stability greatly.

[1] https://search.ci.kubevirt.io/?search=etcdserver%3A+request+timed+out&maxAge=336h&context=1&type=build-log&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

Release note:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.